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PREFACE 


This  report  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  for  the  Office 
of  the  Under  Secretary  for  Defense  (Acquisition)  under  contract  MDA  903  84  C  0031, 
Task  Order  T-G7-535,  issued  1  July  1987,  and  amendments.  The  objective  of  the  task 
was  to  conduct  case  studies  of  ongoing  acquisition  programs  in  order  to  determine  the 
extent  to  which  cost,  schedule,  and  other  predictions  have  been  accurate. 

The  intent  of  this  two-volume  report  is  to  identify  the  characteristics  associated  with 
problems  and  successes  in  the  acquisition  of  tactical  munitions.  Comparisons  of  the 
acquisition  process  between  the  munitions  included  in  the  sample  are  reported  in  Volume  I. 
Volume  II  consists  of  individual  case  studies  of  the  acquisition  process  for  each  of  the 
munitions  included  in  the  sample. 

The  report  was  reviewed  by  Dr.  Richard  E.  Schwartz,  Dr.  John  E.  Hove,  and  Mr. 
Stanley  A.  Horowitz  of  IDA.  Mr.  David  C.  Gogerty,  who  performed  much  of  the  analyses 
for  this  study,  is  an  IDA  consultant 


EXECUTIVE  SUMMARY 


A.  OBJECTIVE  AND  APPROACH 

Contemporary  tactical  munitions  acquisitions  have  generally  suffered  substantial 
technical,  schedule,  and  cost  difficulties  in  attempting  to  achieve  successful  operational 
capabilities.  For  the  most  part,  these  operational  capabilities  have  been  realized  after 
considerable  grief  in  the  development  and  production  processes.  The  Office  of  the  Under 
Secretary  of  Defense  (Acquisition)  requested  this  study  to  examine  the  causes  of  outcomes 
for  selected  tactical  munitions  and  to  document  lessons  learned  from  them.  The  objectives 
of  this  task  were  therefore  to: 

•  Determine  how  substantial  these  difficulties  were  primarily  in  terms  of  cost 
growth  and  schedule  slippage  in  development  and  procurement; 

•  Investigate  how  the  difficulties  originated  and  why  they  seem  to  recur, 

•  Address  what  might  be  done  about  them. 

The  approach  was  to  examine  nineteen  new  and  modified  contemporary  tactical 
munitions  from  thirteen  families,  using  information  from: 

•  Selected  acquisition  reports  and  other  readily  available  literature; 

•  Program  offices  and  contractors. 

A  modified  munition  is  defined  here  as  a  significantly  improved  version  of  an 
operational  munition.  For  example,  the  Sparrow  AIM-7F  was  a  major  modification  in  the 
sequence  of  Sparrow  missiles.  Modification  programs  have  mostly  been  addressed  as 
relatively  straightforward  extensions  of  existing  programs.  However,  tactical  munitions 
modifications  generally  involve  improvements  to  the  guidance  and  control  sections,  and 
these  subsystems  often  represent  two-thirds  to  five-sixths  of  the  total  cost  of  the  munitions, 
so  modification  programs  can  be  very  large. 

The  selection  criteria  for  the  thirteen  families  of  munitions  were: 

•  Still  in  the  inventory  (except  Deadeye  Five-Inch  Guided  Projectile); 

•  Availability  of  Selected  Acquisitions  Report  (SAR)  data; 

•  Mixture  of  air-  and  surface-launched,  intercept,  and  surface  attack; 

•  Representation  by  each  service  and  several  major  contractors. 


The  nineteen  munition  acquisition  programs  are  documented  in  thirteen  case  studies 
in  Volume  n  of  this  paper.  The  outcomes  are  summarized  here  in  Tables  ES-1  and  ES-2. 
Relationships  between  measures  of  the  acquisition  program  outcomes,  measures  of 
program  risks,  types  of  munitions,  and  applicability  of  acquisition  policies  were 
investigated  using  non-parametric  statistical  methods,  and  the  results  are  documented  in 
Volume  I.  The  findings  are  sensitive  to  the  limitation  of  the  database;  sample  size  was  not 
large  enough  to  establish  the  statistical  significance  of  many  of  the  observed  differences  in 
outcomes,  risks,  and  policy  application. 

B.  MAJOR  FINDINGS 

The  analysis  of  the  nineteen  munition  acquisition  programs  showed  that: 

•  Modifications  were  much  less  expensive  to  develop  than  were  new  munitions. 
Average  development  cost  for  modified  munitions  was  about  one-third  of  that 
for  new  munitions  The  highest  actual  development  cost  for  a  modification 
was  less  than  two-thirds  the  average  development  cost  for  new  munitions. 
Among  the  technically  demanding  air-launched  intercept  missiles,  the 
development  cost  for  each  of  the  six  modifications  was  at  least  35  percent  less 
than  the  lowest  of  the  development  costs  for  either  of  the  two  new  missiles. 

•  Development  program  outcomes,  in  terms  of  schedule,  quantity,  and  cost 
growth,  were  on  average  as  bad  or  worse  for  modifications  than  for  new 
munitions.  Difficulties  in  developing  modifications  tended  to  be  greatly 
underestimated.  Development  quantities  for  modified  munitions  increased  by 
an  average  of  83  percent  over  the  initial  estimates,  because  of  unrealistically 
low  initial  estimates  of  test  article  requirements  for  several  air-launched 
intercept  missile  modifications.  Development  quantities  for  new  munitions 
were  initially  estimated  much  more  realistically,  and  subsequently  decreased  by 
an  average  of  16  percent;  large  numbers  of  test  articles  initially  planned  for 
several  surface-launched  surface  attack  munitions  were  later  reduced.  While 
average  development  cost  growth  for  modified  munitions  was  almost  double 
that  for  new  munitions,  the  difference  was  not  statistically  significant. 
Development  schedule  slippage  did  not  differ  significantly  between  new  and 
modified  munitions.  Production  cost  growth  and  total  program  cost  growth 
did  not  vary  significantly  between  new  and  modified  munitions.  These  results 
strongly  suggest  that  acquisition  programs  for  munitions  modifications  are 
equally  as  deserving  of  strict  management  attention  and  procedures  as 
acquisition  programs  for  new  munitions. 


vi 


Table  ES-1.  Summary  of  Acquisition  Program  Schedules,  Costs,  and  Quantities 
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I.U.  -  Information  unavailable. 
N.A.-  Not  applicable. 

*  -  SAR  estimate. 


Table  ES-2.  Summary  of  Acquisition  Program  Outcomes 
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I.D.  =  Insufficient  data. 


•  Development  quantity  growth  and  development  cost  growth  were  significantly 
lower  for  munitions  that  underwent  advanced  development.  However,  it  is 
difficult  to  separate  the  effects  of  advanced  development  from  whether  the 
munition  was  new  or  modified  All  of  the  new  munitions  underwent  advanced 
development,  while  only  two  of  the  modifications  did.  Development  quantity 
growth  and  development  cost  growth  were  lower  for  the  modifications  that 
underwent  advanced  development,  but  the  sample  size  was  too  small  to  show 
that  the  differences  were  statistically  significant 

•  Competition  during  advanced  development  did  not  result  in  statistically 
significant  differences  in  any  of  the  acquisition  program  outcome  measures. 
Competitive  advanced  development  efforts  may  have  been  focused  on  the 
follow-on  FSD  contract  rather  than  on  reducing  technological  risks. 

•  Competition  at  the  subsystem  level  during  full  scale  development  was 
associated  with  lower  development  cost  growth.  None  of  the  munitions  in  the 
sample  had  competition  at  the  system  level  during  full  scale  development 

•  Production  competition  at  either  the  system  or  subsystem  level  had  no 
discemable  effect  on  production  cost  growth. 

•  The  extent  of  the  overlap  between  development  and  production  was  positively 
correlated  with  production  cost  growth. 

•  Production  cost  growth  was  directly  correlated  to  overall  DoD  procurement 
total  obligational  authority  (TO A)  over  the  production  span  for  each  munition. 

•  No  statistically  significant  effects  on  any  of  the  program  outcome  measures 
could  be  discerned  for  independent  testing  or  multi-year  production 
contracting. 

Other  significant  findings  are  summarized  in  the  following  two  sections. 

C .  PROGRAM  OUTCOMES 

Acquisition  program  outcomes  did  not  differ  significantly  between  types  of 
munitions,  between  new  and  modified  munitions,  or  over  time,  except  as  otherwise  noted. 
The  following  significant  outcomes  were  observed: 

•  Total  program  cost  growth  averaged  54  percent,  and  was  highly  correlated 
with  production  cost  growth. 

•  Development  schedules  slipped  by  an  average  of  48  percent,  and  that  slippage 
involved  development  problems  that  subsequently  adversely  affected 
production  costs. 


Development  costs  increased  by  an  average  of  71  percent,  and  that  increase 
was  directly  correlated  with  development  quantity  growth.  Development  cost 
growth  was  not  correlated  with  development  schedule  growth  for  the 
munitions  in  the  sample.  Development  cost  growth  for  intercept  munitions 
was  significantly  higher  than  for  surface  attack  munitions;  several  intercept 
missile  modifications  had  unrealistically  low  initial  development  cost  estimates, 
while  the  initial  development  cost  estimates  for  new  surface  attack  munitions 
were  much  more  realistic. 

Production  costs  for  the  quantities  that  were  originally  estimated  at  the  start  of 
development  increased  by  an  average  of  58  percent.  That  increase  was  directly 
correlated  with  development  schedule  growth,  but  was  not  correlated  with 
growth  in  either  development  costs  or  development  quantities. 

Production  quantities  increased  by  an  average  of  42  percent,  and  that  increase 
was  not  significantly  correlated  with  any  other  program  outcome  measure. 
Production  quantity  growth  increased  during  the  defense  buildup  of  the  early 
1980s. 

Production  stretchout,  defined  as  the  average  increase  in  time  between  delivery 
of  each  unit,  averaged  119  percent.  That  increase  was  not  significantly 
correlated  with  any  of  the  other  program  outcome  measures.  Production 
stretchout  decreased  during  the  defense  buildup  of  the  early  1980s. 

Operational  and  technical  requirements  specified  in  the  SARs  were  fully 
satisfied  by  less  than  half  of  the  munitions  that  had  completed  development. 
Because  some  munitions  were  deployed  without  fully  satisfying  the 
requirements,  the  requirements  may  have  been  unduly  stringent.  Production 
quantity  growth  did  not  differ  significantly  between  the  munitions  for  which 
the  requirements  were  not  satisfied  and  the  munitions  for  which  the 
requirements  were  satisfied. 


D.  EFFECTS  OF  PROGRAM  RISKS 

Information  provided  by  contractors  on  technology  advance  and  resource 
requirements  expectations  and  actual  outcomes  for  eleven  of  the  munition  acquisition 
programs  (see  Volume  I,  Appendix  A  for  questionnaire  and  Volume  II, 
Appendix  A  for  responses)  showed  that: 

•  Development  schedule  growth,  development  cost  growth,  and  production  cost 
growth  were  not  found  to  be  directly  related  to  the  requirements  for  advances 
in  performance,  materials,  or  performance  technologies. 

•  Munitions  that  had  higher  requirements  for  advances  in  production  technology 
also  tended  to  have  higher  percentage  requirements  for  new  test  equipment  and 
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tooling.  Those  munitions  also  tended  to  have  advanced  development, 
independent  cost  estimates  and  low-rate  initial  production  releases  associated 
with  their  acquisitions.  The  munitions  with  high  production  technology 
requirements  that  also  had  advanced  development  and  independent  cost 
estimates,  had  significantly  lower  development  cost  growth  and  development 
quantity  growth  than  did  munitions  with  lower  production  technology 
requirements  but  without  advanced  development  or  independent  cost  estimates. 
However,  the  sample  size  was  not  large  enough  or  sufficiently  stratified  to 
determine  whether  any  effects  of  high  technology  or  resource  risks  may  have 
been  offset  by  the  positive  effects  of  those  acquisition  policies  on  development 
cost  growih  or  development  quantity  growth.  For  similar  reasons,  it  was  not 
possible  to  separate  out  the  effects  on  production  cost  growth  of  high 
technology  and  resource  requirements  from  the  effec  :.  of  low-rate  initial 
production. 

In  general,  contractors  had  difficulty,  even  after  the  fact,  in  quantifying  differences 
between  their  original  program  expectations  and  the  actual  outcomes  in  terms  of  the  extent 
of  the  technology  advance  relative  to  the  state-of-the-art.  Yet  these  same  contractors 
admitted,  qualitatively,  that  several  of  their  programs  were  pushing  the  state-of-the-art  and, 
indeed,  had  suffered  severe  technical,  cost,  or  schedule  problems.  This  dichotomy  points 
out  the  need  for  quantitative  tools  for  assessing  the  technical  state-of-the-art  and  for 
estimating  full  scale  development  (FSD)  scheduling. 

Two  of  the  munitions  (AGM-65D/F/G  IIR  Maverick  and  AGM-88A  HARM)  had 
large  changes  in  the  perceived  nature  of  the  threat  during  full  scale  development  However, 
there  were  no  significant  differences  in  any  of  the  program  outcome  measures  between  the 
munitions  with  changing  threats  and  the  remainder  of  the  munitions  in  the  sample. 

E.  RECURRING  PROBLEMS 

Several  pitfalls  (major  recurring  problems)  that  affected  the  acquisition  program 
outcomes  were  identified  from  *he  case  studies  in  Volume  II: 

•  Development  requirements  for  munition  modification  programs  were  generally 
underestimated; 

•  Technological  uncertainties  were  not  adequately  identified  early  in  the 
development  process; 

•  Design  and  production  concepts  were  inadequately  demonstrated  and  evaluated 
early  in  the  development  process; 

•  Producibility  was  not  adequately  considered  in  design  decisions; 


•  Unrealistic  test  planning  contributed  to  development  schedule  slippage  and 
development  cost  growth; 

•  Larger  overlaps  between  development  and  production  greatly  worsened  the 
effects  of  unforeseen  problems; 

•  Quantitative  methods  for  estimating  development  schedules  and  development 
and  production  costs  were  generally  inadequate,  because  their  scope  was 
inconsistent  or  incomplete,  data  bases  and  analogies  were  inappropriate,  and 
important  subsystems  were  not  shown  separately; 

•  Risks,  schedules,  costs,  and  test  requirements  and  results  were  not 
consistently  subjected  to  independent  review. 

F.  RECOMMENDATIONS 

Actions  that  could  be  taken  to  avoid  the  pitfalls  identified  above,  and  to  improve  the 
outcomes  of  future  tactical  munition  acquisition  programs,  include: 

•  Requiring  major  munition  modification  programs  to  undergo  the  same 
milestone  review  process  as  new  munitions,  including  an  advanced 
development  phase  to  identify  and  control  performance  and  production  risks; 

•  Developing  and  using  technology  trending  models  for  identification  of 
performance  and  production  technological  risks  at  the  initiation  of 
development; 

•  Reducing  technological  risks  through  advanced  development  prototyping  and 
testing  of  critical  subsystems  and  new  production  processes; 

•  Reducing  production  cost  growth  by  requiring  that  producibility  be 
demonstrated  before  approval  of  production  release;  restricting  initial 
production  rates  to  low  levels  until  problems  identified  during  early  operational 
use  can  be  corrected;  and  minimizing  the  extent  of  overlap  between 
development  and  production; 

•  Developing  and  using  more  realistic  schedule  and  cost-estimating  models  and 
procedures; 

•  Providing  for  an  independent  evaluation  by  OSD  staff  of  program  risks, 
schedules,  costs,  and  test  results,  including  consistent  and  full  reporting  in  the 
SARs  (or  DAES),  and  use  of  checklists  of  critical  questions  at  DAB  reviews 
for  new  starts  and  major  modification  programs; 

•  Instituting  a  systematic  effort  to  measure  acquisition  policy  effectiveness  with 
comparisons  across  a  wide  sample  of  acquisition  programs,  and  documenting 
acquisition  program  histories  to  record  lessons  learned. 
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I.  INTRODUCTION 


During  the  past  several  decades,  acquisition  outcomes  for  tactical  munitions 
programs  have  met  with  varying  degrees  of  success  in  terms  of  accomplishing  mission, 
schedule,  and  cost  objectives.  Although  some  programs  have  been  successful,  others  have 
encountered  serious  problems,  in  spite  of  numerous  policy  changes  and  initiatives  intended 
to  improve  the  acquisition  process.  The  differences  in  the  outcomes  of  these  programs 
provide  an  opportunity  to  investigate  the  reasons  for  program  success.  A  better 
understanding  of  the  causes  of  problems  and  the  efficacy  of  acquisition  program  policies 
may  point  to  the  need  for  further  policy  changes. 

A.  OBJECTIVE 

This  paper  examines  the  outcomes  and  lessons  learned  from  selected  tactical 
munitions  programs.  The  objectives  of  this  research  were  to: 

•  Investigate  the  nature,  extent,  and  causes  of  recurring  problems  in  these 
acquisition  programs; 

•  Identify  measures  for  program  outcomes  and  collect  data  for  each; 

•  Present  policy  recommendations  for  dealing  with  the  problems  observed; 

•  Compile  a  series  of  case  studies  that  will  provide  a  corporate  memory  of 
lessons  learned; 

•  Provide  a  checklist  of  major  issues  and  concerns  for  decisionmakers. 

Acquisition  outcomes  were  measured  by  comparing  program  schedule,  cost,  and 
quantity  actuals  to  those  planned  at  the  time  of  the  development  estimate.  We  addressed 
these  outcomes  as  indicators  of  management  success;  we  did  not  undertake  comprehensive 
examination  of  technical  and  performance  measures  relating  to  operational  success. 

B.  APPROACH 

A  sample  of  nineteen  contemporary  tactical  munitions  was  selected  for  analysis. 
The  munitions,  listed  in  Table  1-1,  were  selected  to  represent  different  categories  of 
munitions  (air-  and  surface-launched  intercept  missiles,  air-  and  surface-launched  surface 


attack  missiles  and  rockets,  and  gun-launched  guided  projectiles),  different  service 
managers  (Army,  Navy,  and  Air  Force)  and  varying  degrees  of  success  (including 
cancellation).  Eight  different  companies  were  prime  contractors. 


Table  1-1.  Munitions  Included  in  the  Analysis 


Designator 

Name 

New/  Manage- 
Mod  mens* 

Users* 

Prime 

Second 

Source(s) 

A/RIM-7E/H/J 

Sparrow  DIB  CW 

Mod 

N 

AF,  N,  M,  F 

Raytheon 

Foreign 

AIM-7F 

Sparrow  III  Pulse  Doppler  Mod 

N 

AF,  N,  M,  F 

Raytheon 

General  Dynamics 

A/RIM-7M 

Sparrow  IU  Monopulse 

Mod 

N 

AF,  N,  M,  F 

Raytheon 

General  Dynamics 

AIM-9L 

Sidewinder 

Mod 

N 

AF,  N,  M,  F 

Ford 

Raytheon,  Foreign 

AIM-9M 

Sidewinder 

Mod 

N 

AF,  N,  M,  F 

Ford 

Raytheon 

• 

AIM-54A 

Phoenix 

New 

N 

N,  F 

Hughes 

AIM-54C 

Phoenix 

Mod 

N 

N 

Hughes 

Raytheon 

AIM-120A 

AMRAAM 

New 

AF 

AF,  N,  M,  F 

Hughes 

Raytheon,  Foreign 

FIM-92A 

Stinger-Basic 

New 

A 

A,  AF,  M,  F 

General  Dynamics 

FIM-92A 

Stmger-POST/RMP 

Mod 

A 

A,  AF,  M,  F 

General  Dynamics  Raytheon,  Foreign 

• 

AGM-65D/F/G 

HR  Maverick 

Mod 

AF 

AF,  N,  M,  F 

Hughes 

Raytheon 

A/R/UGM-84 A/C/D  Haipoon 

New 

N 

AF,  N,  F 

McDonnell 

AGM-88A 

HARM 

New 

N 

AF,  N.  M,  F 

Texas  Instruments 

AGM-114A/B 

Hellfire 

New 

A 

A.M.F 

Rockwell,  Martin 

Rockwell,  Martin 

• 

BGM-71A 

TOW  I 

New 

A 

A,  M,  F 

Hughes 

Chrysler,  Foreign 

BGM-71D 

TOWII 

Mod 

A 

A,  M,  F 

Hughes 

Foreign 

- 

MLRS 

New 

A 

A,  F 

LTV.FMC 

Foreign 

M-712 

Copperhead  CLGP 

New 

A 

A,  M,  F 

Martin 

- 

5"  Deadeye  SALGP 

New 

N 

N 

Martin 

• 

*A  -  Anny 
AF  -  Air  Force 
N  -Navy 
M  -Marina 
F  -Foreign 

• 

In  order  to  investigate  the  extent  of  any  differences  in  acquisition  program  outcomes 
between  new  and  modified  munitions,  the  sample  contains  ten  new  munitions  and  nine 
modifications  of  existing  munitions.  Two  of  the  new  munitions  (Harpoon  and  Hellfire) 
included  minor  modifications.  ^ 

The  sample  is  also  spread  over  about  30  years.  The  AIM-7E  Sparrow  began 
engineering  development  in  January  1960,  and  a  modified  version  is  still  being  produced 
three  decades  later.  The  AMRAAM  began  full  scale  development  in  December  1981.  The 
sample  is  contemporary  in  that  all  but  one  of  the  munitions  are  either  still  in  production  or  • 
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in  service,  or  are  previous  versions  of  munitions  still  in  production  or  in  service.  One  of 
the  munitions  (5"  Deadeye  SALGP)  never  entered  production  and  was  not  deployed. 
Production  has  been  completed  for  six  of  the  munitions  while  the  remaining  twelve  are  still 
being  produced.  Timelines  of  the  development  and  production  phases  for  the  munitions  are 
shown  in  Figure  1-1;  bullets  indicate  the  start  of  each  phase.  The  figure  does  not  show 
concurrency  or  gaps  between  the  development  and  production  phases.  In  actuality,  several 
of  the  acquisition  programs  had  overlaps  between  development  and  production  phases, 
while  other  programs  had  gaps  between  phases. 

For  each  of  the  munitions  included  in  the  sample,  schedule  dates,  cost,  and 
production  quantity  data,  and  narrative  information  were  obtained  from  Selected 
Acquisitions  Reports  (SARs),  the  latest  available  editions  of  references  [1, 2,  and  3],  and 
interviews  with  program  management  and  contractor  personnel.  The  SARs  were  used  as 
the  primary  source  of  information  because  they  are  official  government  documents  and  are 
readily  available. 

Original  estimates  (generally  as  of  Milestone  II)  of  schedules,  costs,  and  quantities 
were  obtained  from  the  earliest  available  SAR  for  each  munition.  Because  some  of  the 
acquisition  programs  predate  the  initiation  of  SARs  in  1967,  their  development  estimates  of 
schedules,  costs,  and  quantities  shown  in  this  report  may  not  have  been  the  true  original 
development  estimates;  they  may  instead  be  subsequent  revisions.  In  addition,  revised 
estimates  of  schedules,  costs,  and  quantities  were  obtained  from  all  of  the  year-end  SARs 
following  the  first  available  SAR  for  each  program.  The  December  1987  SAR  was  the 
latest  available. 

Using  these  data,  calculations  were  made  to  show  the  extent  of  schedule  and  cost 
growth  (or  shrinkage  in  a  few  cases)  for  each  program.  Because  production  quantities 
have  changed  substantially  since  the  original  development  estimates  for  most  of  the 
programs,  and  because  these  quantity  changes  affect  cost  changes,  the  extent  of  cost 
growth  was  measured  by  using  the  current  estimate  cumulative  cost  curve  to  calculate  what 
the  program  cost  would  have  been  if  the  development  estimate  quantity  had  been  produced. 

To  obtain  information  on  technological  risks  and  new  resource  requirements,  IDA 
sent  questionnaires  to  the  prime  contractors  of  the  munitions  in  the  sample.  The 
questionnaire  is  reproduced  in  Appendix  A.  Responses  were  received  from  four  of  the 
prime  contractors  for  eleven  of  the  munitions  in  the  sample. 
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Figure  1-1.  Contract  Timelines 


In  addition  to  the  quantitative  data,  narrative  information  was  obtained  on  the 
applicability  of  various  defense  acquisition  policies  and  initiatives  to  the  munitions. 
Information  was  also  obtained  where  available  concerning  stability  of  the  threat;  the  extent 
to  which  the  performance  and  technical  requirements  were  satisfied;  the  stability  of 
development  and  production  funding;  and  the  nature,  extent,  and  cause  of  major  problems 
encountered  and  how  they  were  managed.  This  narrative  information  and  the  quantitative 
data  were  used  to  prepare  thirteen  separate  case  studies  for  the  nineteen  munitions  in  the 
sample.  These  studies  are  contained  in  Volume  II  of  this  report.  Those  case  studies 
contain  the  quantitative  data  and  narrative  information  to  support  the  conclusions  and 
recommendations  presented  in  this  volume. 

Volume  I  contains  the  results  of  a  cross-section  study  of  the  differences  in  program 
outcome  variables  between  the  munitions  in  the  sample.  Non-parametric  statistical  methods 
were  used  in  the  analysis  of  the  quantitative  data.  The  distributions  of  the  quantitative 
measures  of  program  outcomes,  resource  requirements,  funding  stability,  and  concurrency 
were,  in  general,  highly  skewed  and  irregularly  shaped.  They  were  not  normally 
distributed  with  bell-shaped  frequencies,  nor  could  they  be  easily  transformed  into  the 
normal  bell-shaped  frequencies.  The  statistical  significance  of  differences  in  a  quantitative 
measure  between  categories  of  munitions  characteristics,  levels  of  technological  risk,  or 
applicability  of  various  acquisition  policies,  was  evaluated  using  Kruskal-Wallis  and  Mann- 
Whitney  tests. 

Spearman  rank  correlations  and  their  levels  of  significance  were  used  to  measure 
the  relationships  between  program  outcomes,  resource  requirements,  funding  stability,  and 
concurrency  measures.  Correlations  of  subsamples  of  these  quantitative  measures  for 
particular  categories  of  munitions  characteristics,  levels  of  technological  risk,  or 
applicability  of  various  acquisition  policies,  were  generally  not  evaluated  because  of  the 
small  numbers  o."  munitions  in  each  category.  Correlations  or  differences  which  are  stated 
to  be  significant  in  the  text  in  this  volume  have  a  statistical  significance  level  of  10  percent 
or  less. 

C .  OUTLINE  OF  THE  REPORT 

This  report  addresses  program  outcomes  and  risks  in  the  context  of  several 
acquisition  strategies,  program  risks,  and  program  characteristics.  Figure  1-2  shows  the 
chapter  where  each  category  of  data  is  analyzed.  The  arrows  in  Figure  1-2  show  which 
other  categories  of  data  are  used  in  the  analysis  of  a  particular  category.  Basic  program 


characteristics,  in  terms  of  type  of  munition,  the  services  and  prime  contractors 
responsible,  whether  or  not  the  munition  was  new  or  a  modification,  and  when  the 
munition  entered  development  ana  production  were  discussed  in  the  preceding  section. 


Figure  1*2.  Data  Categories  and  Evaluation  Relationships 


Chapter  II  describes  and  compares  the  acquisition  strategies  for  the  munitions  in  the 
sample  and  relates  them  to  program  characteristics.  Chapter  in  does  the  same  for  program 
risks.  Relationships  between  program  outcomes  and  acquisition  strategies  are  analyzed  in 
Chapter  IV.  Whether  or  not  the  program  outcome  measures  are  correlated  to  each  other  or 
when  development  or  production  started,  or  to  resource  requirements,  or  to  the  measures  of 
funding  stability  or  the  overlap  between  development  and  production  is  also  reported  in 
Chapter  IV.  Significant  differences  between  munitions  to  which  a  particular  acquisition 
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policy  was  applied  and  the  other  munitions  are  reported  in  Chapter  IV.  Similar  evaluations 
are  shown  in  Chapter  IV  for  subsamples  categorized  by  munitions  characteristics,  levels  of 
technological  risk,  and  whether  or  not  the  design  requirements  were  satisfied. 
Conclusions,  in  terms  of  lessons  learned,  and  recommendations  to  assist  in  identifying  and 
managing  potential  acquisition  problems  are  contained  in  Chapter  V.  The  statistical 
malyses  and  supporting  data  for  those  analyses  and  for  the  figures  and  tables  in  the  text, 
arc  documented  in  the  Notes  following  Chapter  V. 

The  second  volume  of  this  report  consists  of  thirteen  separate  case  studies  covering 
the  nineteen  munitions  included  in  the  sample.  Each  case  study  contains  a  program 
description,  covering  the  mission,  history,  and  acquisition  strategy;  a  system  technical 
description,  covering  the  munition  subsystems,  performance  and  technical  requirements 
and  test  results;  a  program  cost/schedule  assessment  covering  cost/schedule  estimates, 
program  funding,  and  cost/quantity  relationships;  and  lessons  learned,  in  terms  of  the 
threat,  requirements,  technology,  acquisition  policy,  and  political  environment. 


D .  RELATED  WORK 

During  the  course  of  our  work,  we  reviewed  related  research  in  the  field  [4  through 
43].  We  found  our  findings  agreed  with  previous  research  in  some  cases  and  disagreed  in 
others. 

Our  findings  agree  with  some  of  the  reasons  for  program  success  identified  in  [4 
and  5],  and  our  recommendations  are  consistent  with  recommendations  in  those  works  and 
in  [6  and  7].  In  particular,  the  importance  of  identifying  and  controlling  technological 
risks,  and  the  findings  on  the  adverse  effects  of  poor  cost  estimating,  as  cited  in  those 
references,  are  fully  supported  here.  Our  work  supports  the  recommendations  made  in  [8] 
for  identifying  technological  risks  by  quantifying  technological  trends  over  time.  Our 
analyses  also  support  the  recommendations  made  over  thirty  years  ago  in  [9]  for  reducing 
technological  risks  through  advanced  development  and  prototyping. 

Problems  of  producibility  cited  in  [6]  were  identified  from  an  examination  of  six 
weapons,  of  which  two  (HARM  and  Copperhead)  are  included  in  our  sample.  Many  of  the 
same  problems  can  be  seen  in  munitions  in  our  sample  that  were  not  considered  in  [6].  The 
same  recommendations  for  demonstrating  producibility  during  development  are  made  here. 

On  the  other  hand,  our  analysis  does  not  include  some  of  the  other  reasons  for 
program  success  identified  in  [4,  5,  6,  and  7].  We  did  not  directly  consider  quality  of 
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program  management  or  the  contractor  office  and  technical  staffs,  as  did  [4  and  5], 
although  inferences  can  be  drawn  from  the  acquisition  program  outcome  measures 
evaluated  here.  Nor  do  our  results  show  that  stability  of  requirements  and  funding  are 
major  reasons  for  program  success,  as  cited  in  [4,  5, 6,  and  7], 

This  report  is  the  latest  in  a  series  of  projects  at  IDA  that  compared  schedule  and 
cost  outcomes  of  a  number  of  weapons.  Those  earlier  projects  [10  and  11]  examined  a 
wider  variety  and  number  of  weapons  than  reported  on  here,  and  their  findings  and 
recommendations  are  similar  to  ours.  This  report  is  based  on  a  more  detailed  examination 
of  fewer  munitions  and  uses  more  recent  data  than  were  available  for  [10  and  11]. 
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n.  THE  ACQUISITION  ENVIRONMENT 
AND  STRATEGY 


This  chapter  compares  how  defense  acquisition  policies  and  initiatives  woe  applied 
to  the  munitions  in  the  sample.  To  provide  a  context  for  these  comparisons,  the  chapter 
opens  with  a  brief  description  of  the  existing  acquisition  milestone  process  and  how  it  has 
evolved  since  1960. 

A.  DEFENSE  ACQUISITION  MILESTONES 

The  purposes  of  the  defense  acquisition  milestone  process  are: 

•  To  evaluate  the  progress  of  an  acquisition  program  at  specific  milestones; 

•  To  ensure  that  appropriate  requirements  are  met  at  each  milestone  before  the 
acquisition  program  is  allowed  to  proceed 

The  phases  of  the  process  according  to  Department  of  Defense  (DoD)  Directive 
5000.1  and  DoD  Instruction  5000.2  (dated  March  12, 1986),  are  displayed  in  Figure  II- 1. 
The  figure  also  presents  information  about  documentation  and  the  activities  that  must  be 
accomplished  before  each  milestone  review. 

The  milestone  dates  and  initial  operational  capability  (IOC)  dates  of  the  munitions  in 
our  sample  are  shown  in  Table  II- 1.  These  dates  are  the  latest  estimates  from  the  SARs  for 
the  different  munitions,  as  summarized  in  the  case  studies  in  Volume  II.  Where  no 
milestone  dates  are  shown,  either  the  acquisition  phase  of  the  program  predated  the 
definition  and  establishment  of  a  required  reporting  milestone,  or  the  milestone  dates  were 
not  recorded  completely  in  the  SARs.  The  SARs  themselves  were  established  in  1967,  and 
Milestones  I  (Concept  Selection),  II  (Program  Go-ahead),  and  III  (Production  Approval) 
were  established  in  1970  to  allow  the  Office  of  the  Secretary  of  Defense/Director  of 
Defense  Research  and  Engineering  (OSD/DDR&E)  to  monitor  major  programs  at  specific 
major  decision  points.  Detailed  program  direction  and  management  was  to  be  provided  by 
the  services,  and  a  Development  Concept  Paper  (DCP)  was  to  serve  as  a  contract  between 
OSD/DDR&E  and  the  service.  By  that  time,  some  of  the  munitions  in  the  sample  had 
already  begun  advanced  development  or  full  scale  (engineering)  development 


Defense  Acquisition  Process  Phases  (Prior  to  1988) 


Table  IM.  Milestone  and  Initial  Operational  Capability  Dates 
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N.A.  =  Not  Applicable. 
l.U.  »  Information  unavailable. 


In  197S,  the  service  secretaries  were  given  authority  for  production  decisions  made 
after  Milestone  HI.  In  1979,  Milestone  m  was  separated  the  Milestones  IHA  and  mB,  as 
shown  in  Figure  II- 1.  This  change  occurred  after  most  of  the  munitions  in  the  sample  had 
entered  production.  Milestone  0  (Mission  Need  Determination)  was  established  after  most 
of  the  munitions  in  the  sample  had  entered  concept  exploration  or  a  later  acquisition  phase. 
Milestones  IV  (Logistics  Readiness  Review)  and  V  (Major  Upgrade  Decision)  were 
established  in  September  1987,  but  have  not  been  applied  to  any  of  the  munitions  in  the 
sample. 

The  reason  no  Milestone  II  dates  are  given  for  the  AIM-54A  Phoenix,  Milestone 
TTTA  or  TTTB  dates  for  AIM/RIM-7E/H/J  Sparrow,  or  Milestone  HLA  dates  for  AIM-54A 
and  AIM-54C  Phoenix  and  HARM  is  because  the  milestones  were  established  too  late  to  be 
applicable  to  those  programs.  The  reason  that  there  were  no  Milestone  IHA  dates  for 
AIM-7F  Sparrow,  AIM-9L  Sidewinder,  TOW  I,  or  TOW  n  is  that  there  were  no  separate 
low-rate  initial  production  phases  for  those  programs. 

Two  anomalies  can  be  observed  in  Table  II- 1.  The  first  is  that  most  of  the  munition 
modification  programs  in  the  sample  were  allowed  to  proceed  to  Milestone  II  (Program  Go- 
ahead)  without  a  Milestone  I  decision  to  identify  and  eliminate  significant  performance, 
production,  schedule,  and  cost  uncertainties. 

The  second  anomaly  is  the  missing  Milestone  III  dates  for  AIM-7F  Sparrow 
and  Harpoon,  and  the  missing  Milestone  IIIA  date  for  AIM/RIM-7M  Sparrow.  These 
missing  dates  can  only  be  explained  by  either:  (1)  incomplete  recording  in  the 
SARs,  or  (2)  the  milestone  decisions  never  took  place. 

B.  ACQUISITION  STRATEGY 

According  to  OMB  Circular  A- 109,  which  governs  major  acquisitions,  an 
acquisition  strategy  should  be  developed  "as  soon  as  the  agency  decides  to  solicit 
alternative  system  design  concepts  that  could  lead  to  the  acquisition  of  a  new  major 
system."  The  elements  to  be  considered  in  developing  an  acquisition  strategy  can  be  put 
into  three  categories: 

«  Strategic  concerns,  including  national  objectives,  the  threat,  operational 
requirements,  the  technology  base,  overall  program  objectives,  market  factors, 
and  the  critical  program  issues; 

•  Technical  concerns,  specifically  relating  to  design,  test  and  evaluation, 
production,  and  deployment; 


•  Resource  concerns,  including  personnel/organization,  schedule,  busi¬ 
ness/financial,  management  information,  and  facilities. 

For  elements  critical  to  the  success  of  the  program,  alternatives  and  decision  time 
intervals  (windows)  must  be  selected  that  meet  program  objectives.  These  alternatives  and 
windows  together  constitute  the  acquisition  strategy.  A  contemporary  acquisition  strategy 
would  include: 

•  Estimates  for  cost,  schedule,  and  performance  during  the  entire  process; 

•  A  contracting  plan  that  includes  plans  for  competition; 

•  An  assessment  of  the  type  of  warranties  that  will  be  sought; 

•  Plans  and  criteria  for  source  selection  at  each  stage; 

•  Funding  plans; 

•  Plans  for  testing  and  evaluation,  possibly  including  prototypes; 

•  Plans  for  development  and  allocation  of  appropriate  logistics  support; 

•  An  assessment  of  program  risks. 

It  is  not  possible  to  reconstruct,  exactly  and  totally,  the  acquisition  strategy  for  each 
of  the  munitions  in  the  sample.  All  that  is  available  is  information  showing  whether  or  not 
certain  policies  were  incorporated  into  the  acquisition  strategy  for  each  munition.  In  the 
following  sections  of  this  chapter,  each  munition  is  characterized  in  terms  of  the  different 
acquisition  policies  and  initiatives  applied  during  each  phase  of  the  acquisition  process. 

One  of  the  important  distinctions  among  the  munitions  in  the  sample  concerns  the 
extent  of  competition,  which  has  become  more  important  with  the  various  DoD  competitive 
initiatives  and  the  Competition  in  Contracting  Act  The  level  of  competition  can  vary  from 
none  at  all  (sole  source),  to  competition  over  design  and  price  during  the  development 
phases,  to  competition  over  price  only  during  the  production  phases.  The  applicability  of 
these  different  levels  of  competition  to  the  munitions  in  the  sample  is  shown  in  Figure  II-2. 

C.  ACQUISITION  POLICIES  DURING  ADVANCED  DEVELOPMENT 

Application  of  selected  acquisition  policies  and  initiatives  during  the  demonstration 
and  validation  phase  (also  referred  to  as  "advanced  development")  are  shown  in  Table  n-2 
for  each  of  the  munitions  in  the  sample. 


Figure  11-2.  Types  of  Competition 


Table  11-2.  Advanced  Development  Start  Dates  and  Acquisition  Policies 
Applied  During  Advanced  Development 


N.A.  -  Not  applicable. 

I.U.  -  Information  unavailable. 


Of  the  nine  modification  programs,  only  the  AIM/RIM-7M  Monopulse  and  the  HR 
Maverick  programs  had  advanced  development  phases,  but  the  advanced  development 
contract  dates  for  the  UR  Maverick  are  not  readily  available,  and  no  Milestone  I  dates  are 
shown  in  the  SAR  for  either  munition. 

All  of  the  new  munitions  in  the  sample  had  advanced  development  phases. 
However,  for  the  Basic  Stinger  and  the  5"  Deadeye  SALGP,  the  advanced  development 
contract  dates  are  not  readily  available,  and  no  Milestone  I  dates  are  shown  in  the  SARs. 

Of  the  nine  munitions  for  which  the  dates  of  entering  advanced  development  could 
be  determined,  the  six  that  entered  advanced  development  after  1971  all  had  competition 
during  the  advanced  development  phase.  All  of  these  nine  had  either  whole  system  or 
subsystem  prototypes,  regardless  of  when  they  entered  advanced  development 

Of  the  ten  new  programs,  six  were  competitive  during  the  advanced  development 
phase,  and  all  six  of  those  plus  two  non-competitive  programs  had  whole  system 
prototypes.  There  is  not  enough  evidence  to  establish  any  relationship  between  prototyping 
and  competition;  however,  program  managers  and  contractors  for  the  new  munitions  that 
underwent  competition  seemed  to  realize  the  value  of  prototyping. 

Of  the  two  modification  programs  that  underwent  advanced  development,  one  was 
competitive,  and  neither  had  whole  system  prototypes,  but  both  had  subsystem  prototypes. 

There  were  no  clear  relationships  between  whether  or  not  the  program  had  an 
advanced  development  phase  and: 

•  The  program  manager; 

•  The  prime  contractor, 

•  The  type  of  munition. 

D.  ACQUISITION  POLICIES  DURING  FULL  SCALE  DEVELOPMENT 

Application  of  selected  acquisition  policies  during  the  full  scale  development  phase 
are  shown  in  Table  II-3.  Limited  data  are  shown  for  the  MLRS,  which  had  a  tailored 
acquisition  program  that  proceeded  directly  from  prototype  construction  and  testing  to  low- 
rate  initial  production  without  a  full  scale  development  phase. 

Although  none  of  the  munitions  in  the  sample  had  competition  at  the  system  level 
during  full  scale  development,  four  had  competition  at  the  subsystem  level.  Eight  different 


combinations  of  contract  types  were  used.  Only  three  of  the  munitions  were  subjected  to 
independent  tests,  while  six  had  independent  cost  estimates.  The  type  (or  combination  of 
types)  of  contract(s)  and  whether  or  not  there  was  competition  at  the  subsystem  level, 
independent  testing,  or  independent  cost  estimates  are  not  related  to  when  the  munitions 
entered  full  scale  development.  Further,  there  were  no  clear  relationships  among  the 
various  acquisition  policies  applied. 

There  are,  however,  clear  relationships  between  three  of  the  acquisition  policies 
applied  and  whether  or  not  there  was  an  advanced  development  phase.  Of  the  seven 
munitions  (all  modifications)  that  did  not  have  advanced  development,  none  had 
competition  at  the  subsystem  level,  independent  testing,  or  independent  cost  estimates.  Of 
the  twelve  munitions  that  did  have  advanced  development,  (ten  new  and  two  modified), 
four  had  competition  at  the  subsystem  level,  three  had  independent  testing,  and  six  had 
independent  cost  estimates.  It  appears  that  munitions  that  underwent  advanced 
development  were  more  likely  to  have  competition  at  the  subsystem  level,  independent 
testing,  and  independent  cost  estimates.! 

There  were  no  clear  relationships  between  the  application  of  any  of  the  acquisition 
policies  during  full  scale  development  and: 

•  The  program  manager, 

•  The  prime  contractor, 

•  The  type  of  munition. 

E.  ACQUISITION  POLICIES  DURING  PRODUCTION 

Application  of  selected  acquisition  policies  during  the  production  phase  are  shown  in  Table 
n-4  for  each  of  the  munitions  in  the  sample.  No  data  are  shown  for  the  5"  Deadeye 
SALGP,  which  was  cancelled  just  before  it  was  due  to  enter  production.  Further,  although 
whole  system  competition  is  shown  in  Table  H-4  for  the  MLRS,  that  was  a  "winner  take 
all"  competition,  which  was  decided  at  the  completion  of  advanced  development 

Nine  of  the  munitions  in  the  sample  had  low-rate  production  prior  to  entering  full- 
rate  production.  Two  of  the  munitions  (AIM-54C  Phoenix  and  AMRAAM)  have  still  not 
entered  full-rate  production.  Twelve  of  the  munitions  in  the  sample  have  had,  or  will  have, 
production  competition  at  the  system  level,  and  ten  had  production  competition  at  the 
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Table  11-4.  Production  Start  Dates  and  Acquisition  Policies  Applied  During  Production 
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subsystem  level.  Eight  different  combinations  of  contract  types  were  used.  Five  of  the 
munitions  have  had,  or  will  have,  multi-year  production  contracts,  and  seven  have  had,  or 
will  have,  foreign  production.  There  are  warranties  at  the  system  level  for  three  of  the 
munitions  and  at  the  subsystem  level  for  seven  of  the  munitions. 

With  the  exception  of  competition  at  the  subsystem  level,  none  of  the  acquisition 
policies  seem  to  be  related  to  when  production  began.  Munitions  that  entered  production 
later  in  the  period  covered  by  the  sample  were  much  more  likely  to  have  production 
competition  at  the  subsystem  level  than  were  munitions  that  entered  production  earlier .2 

Only  one  clear  relationship  was  found  among  the  various  acquisition  policies 
applied  during  production.  The  relationship  between  production  competition  at  the  system 
and  subsystem  levels  is  shown  in  Table  II-5.  Whether  or  not  there  was  production 
competition  at  the  system  level  appears  to  be  closely  related  to  whether  or  not  there  was 
production  competition  at  the  subsystem  level.  3 


Table  11-5.  Relationship  Between  Production  Competition 
at  the  System  and  Subsystem  Levels 


Whether  or  not  the  system  is  new  or  modified  does  not  have  any  clear  relationship 
to  the  type  of  production  contract,  or  whether  or  not  there  were  multi-year  contract,  system 
or  subsystem  warranties,  or  foreign  production.  However,  there  are  clear  relationships 
between  whether  the  munition  was  new  or  modified  and  whether  or  not  there  was 
subsystem  competition  and  low-rate  production  before  entering  full-rate  production.4 
Seven  of  the  nine  new  munitions  that  entered  production  had  an  initial  low- rate  production 
period,  in  contrast  to  only  two  of  the  eight  modified  munitions  for  which  data  were 
available.  Eight  of  the  nine  modified  munitions  had  production  competition  at  the 


subsystem  level,  in  contrast  to  two  of  the  eight  new  munitions  for  which  data  were 
available. 

With  the  exception  of  multi-year  contracting,  there  were  no  clear  relationships 
between  application  of  any  of  the  acquisition  policies  and: 

•  The  program  manager, 

•  The  prime  contractor, 

•  The  type  of  munition. 

There  were  clear  relationships  between  multi-year  contracting,  which  service  is 
prime  manager,  and  the  type  of  munition.  5  Five  of  the  seven  Army  munitions  that  entered 
production  had  multi-year  contracts,  in  contrast  to  none  of  the  nine  Navy  munitions  and 
neither  of  the  two  Air  Force  munitions.  None  of  the  eight  air-launched  intercept  missiles 
(AIM  designators)  and  none  of  the  four  air-launched  surface  attack  missiles  (AGM 
designators)  that  entered  production  had  multi-year  contracts,  in  contrast  to  five  of  the  six 
other  munitions  in  the  sample. 


m.  COMPARISONS  OF  ACQUISITION  PROGRAM  RISKS 


In  the  previous  chapter,  the  nineteen  munitions  in  the  sample  were  compared  in 
terms  of  the  application  of  different  major  acquisition  policies  that  were  introduced  since 
1960.  In  this  chapter,  the  nineteen  munitions  are  compared  in  terms  of  the  various  risks 
that  were  faced  during  acquisition.  The  specific  risks  included  in  the  comparisons  are: 

•  Changes  in  the  threat; 

•  Extent  of  technology  change  required  to  meet  performance  and  production 
requirements; 

•  Extent  of  requirements  for  new  resources; 

•  Stability  of  funding; 

•  Extent  of  overlap  between  development  and  production; 

•  Availability  of  required  complementary  systems  such  as  platforms  or  target 
designators. 

A.  CHANGES  IN  THE  THREAT 

Changes  in  the  threat  to  be  combatted  by  a  munition  affect  the  acquisition  program 
for  that  munition  in  two  possible  ways:  changes  in  performance  requirements  and  changes 
in  the  quantity  to  be  acquired.  Changes  in  the  quantity  requirements  can  result  from 
changes  in  the  numbers  of  targets  or  from  choices  between  munitions  based  upon  budget 
availability  and  differences  in  cost  and  performance.  However,  the  selected  acquisition 
reports  (SARs)  do  not  always  show  the  specific  reasons  for  all  of  the  program  quantity 
changes.  For  successful  acquisitions,  the  program  quantities  tend  to  increase  beyond  the 
numbers  initially  projected,  as  will  be  shown  in  the  next  chapter. 

Changing  performance  requirements  during  FSD  occurred  for  two  of  the  nineteen 
munitions  in  the  sample:  IIR  Maverick  and  HARM.  Performance  requirements  for  the 
imaging  infra-red  (IIR)  seeker  for  the  Maverick  were  changed  as  a  result  of  intelligence 
information  on  the  levels  of  contrast  between  potential  targets  in  potential  target 
environments.  Performance  requirements  for  the  HARM  were  changed  substantially  as  a 
result  of  intelligence  information  on  the  emission  characteristics  of  potential  targets.  The 


extent  of  any  increase  in  development  costs  or  delay  in  the  development  schedule  due  to 
changes  in  performance  requirements  cannot  always  be  separated  in  the  SARs  from  other 
changes  in  development  costs  or  schedules. 


Both  the  EtR  Maverick  and  HARM  programs  included  many  of  the  acquisition 
policies  described  in  Chapter  H  Both  had  advanced  development  phases  in  which  either 
system  or  subsystem  prototypes  were  produced.  The  advanced  development  phase  was  # 

competitive  for  one  of  the  munitions  (HARM),  but  full  scale  development  was  not 
competitive  for  either.  Both  of  these  munitions  had  independent  testing,  independent  cost 
estimates,  low-rate  initial  production  phases,  competition  during  production  at  both  the 
system  (projected  for  the  HARM)  and  subsystem  levels,  and  system  warranties.1  The  • 

HARM  also  had  warranties  at  the  subsystem  level.  This  pattern  differed  from  the 
application  of  the  acquisition  policies  to  the  other  munitions  in  the  sample  in  that: 

•  Only  one  of  the  other  munitions  was  subjected  to  independent  testing; 

•  Only  four  of  the  other  munitions  had  independent  cost  estimates;  ® 

•  Only  four  of  the  other  munitions  had  system  warranties. 


B .  REQUIREMENTS  FOR  ADVANCES  IN  TECHNOLOGY 

The  prime  contractors  for  eleven  of  the  munitions  in  the  sample  provided 
information  on  the  expected  level  of  technical  advance  required  at  the  time  the  development 
program  was  initiated,  and  the  actual  level  of  technical  advance  required  to  complete  the 
development  program.  This  analysis  was  restricted  to  the  munitions'  guidance  subsystems 
because: 

•  The  guidance  system  was  usually  responsible  for  the  largest  subsystem 
development  costs, 

•  Many  of  the  other  subsystems  involve  numerous  subcontractors  from  whom 
data  could  be  obtained  only  with  a  much  larger  expenditure  of  time  and  effort 
than  was  available  for  this  project 


Each  of  the  contractors  for  the  eleven  munitions  submitted  data  in  reply  to  a 
questionnaire  on  the  level  of  technological  advance  required  to  meet  performance  and 
technical  requirements  and  the  extent  of  usage  of  new  or  unfamiliar  (to  the  contractor) 
materials  and  new  or  unfamiliar  production  processes.  For  each  of  these  three  measures  of 
technical  advances,  the  contractors  submitted  responses  showing  whether  the  requirement 
was  satisfied  by: 
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•  Off-the-shelf  technology; 

•  Minor  technological  changes  (less  than  10  percent); 

•  Substantial  technological  changes  (between  1 1  percent  and  SO  percent); 

•  Extensive  technological  changes  (greater  than  50  percent); 

•  All  new  technology. 

All  new  technology,  in  terms  of  performance,  materials,  or  production  processes,  was 
required  for  none  of  the  eleven  munitions  for  which  responses  were  obtained. 

The  questionnaire  used  to  obtain  this  information  is  reproduced  in  Appendix  A. 
The  responses  to  the  questionnaire  are  contained  in  Volume  II.  Because  the  responses  are 
proprietary,  the  data  presented  here  are  aggregated  so  as  to  be  unidentifiable  as  to  contractor 
or  munition. 

The  required  levels  of  advance  in  technology  do  not  show  any  particular  pattern 
over  time  when  the  munitions  are  arranged  chronologically  according  to  full  scale 
development  start  date.  The  chronology  is  shown  in  Figure  HI-1. 

Requirements  for  advances  in  performance,  materials,  or  production  processes 
were  not  significantly  related  to  munition  categories.2  Differences  between  the  required 
levels  of  advance  in  technology  and  munition  categories  are  shown  in  Table  ID-1. 

Even  though  five  of  the  seven  new  munitions  required  extensive  advances  in 
performance,  in  contrast  to  one  of  the  four  modified  munitions,  the  difference  is  not 
statistically  significant  Substantial  or  extensive  advances  in  production  process  technology 
were  required  by  all  seven  of  the  new  munitions,  but  by  only  two  of  the  four  modified 
munitions;  this  difference  is  not  quite  statistically  significant 

Advances  in  performance  did  not  always  require  comparable  advances  in  materials 
or  production  technologies  for  the  munitions  in  the  sample.3  However,  the  levels  of 
required  advances  in  materials  and  production  technologies  were  the  same  for  six  of  the 
new  munitions  but  for  only  one  of  the  modified  munitions,  as  shown  in  Table  III-2.  This 
result  for  new  munitions  should  not  be  surprising.  Advances  in  materials  technology  for 
new  munitions  generally  require  advances  in  production  processes.  For  modified 
munitions,  there  is  more  likely  to  have  already  been  some  experience  with  either  the 
materials  or  production  technologies  to  be  used. 
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Performance  Advance  Chronology 
Extensive  •  • 

Substantial 
Minor 


Material  Advance  Chronology  ■ 


Extensive 

Substantial 

Minor 

Off-the-Shelf 


Production  Advance  Chronology - —  ► 

Extensive 
Substantial 
Minor 

Figure  111-1.  Required  Advance  In  Technology  in  Chronological  Order 
of  the  Start  of  Full  Scale  Development 

Whether  or  not  certain  of  the  acquisition  policy  initiatives  were  applied  was  related 
to  the  extent  of  technological  advance  required  for  the  munitions  in  the  sample.4  The 
numbers  of  munitions  at  each  level  of  required  technological  advance  to  which  the 
acquisition  policy  initiatives  were  applied  are  shown  in  Table  HI-3. 


Advanced  development  was  used  as  a  means  for  alleviating  problems  of 
technological  advancement  tor  most  of  the  munitions  in  the  sample.  Five  of  the  six 
munitions  with  extensive  requirements  for  advances  in  performance  underwent  an 
advanced  development  phase.  Four  of  those  five  had  competitive  advanced  development 
phases  and  prototypes.  Seven  of  the  nine  munitions  that  had  either  substantial  or  extensive 
requirements  for  advances  in  materials,  and  eight  of  the  nine  munitions  that  had  either 
substantial  or  extensive  requirements  for  advances  in  production  process  technologies  also 
had  advanced  development  phases.  Those  munitions  with  substantial  or  extensive 
requirements  for  advances  in  production  technologies  were  significantly  more  likely  to  have 
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had  advanced  development  phases  than  were  munitions  with  lesser  requirements  for 
advances  in  production  technologies.5 


Table  IIM.  Differences  Between  Program  Characteristics 
and  Requirements  for  Advances  in  Technology 
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Table  111*2.  Numbers  of  Munitions  for  Each  Level 
of  Required  Material  and  Production  Technology 
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Competition  was  generally  not  used  as  a  means  of  providing  alternative  solutions  to 
problems  of  technological  advancement  during  full  scale  development.  None  of  the 
munitions  had  competition  at  the  systems  level  during  engineering  development,  regardless 
of  the  extent  of  required  advances  in  technology.  The  extent  of  competition  at  the 
subsystem  level  during  engineering  development  was  not  much  greater. 

Munitions  with  substantial  or  extensive  requirements  for  advances  in  technology 
were  less  likely  to  have  had  independent  testing.  However,  independent  cost  estimates 
were  used  in  attempting  to  control  the  costs  associated  with  technological  advancement. 
Those  munitions  with  substantial  or  extensive  requirements  for  advances  in  production 
technologies  were  significantly  more  likely  to  have  had  independent  cost  estimates  than 
were  munitions  with  lesser  requirements  for  advances  in  production  technologies.^ 

Low-rate  initial  production  phases  were  also  used  in  attempting  to  control  the  costs 
associated  with  technological  advancement.  This  was  true  for  six  of  the  nine  munitions 
with  substantial  or  extensive  advances  in  performance,  five  of  the  eight  with  substantial  or 


extensive  advances  in  materials  technology,  and  six  of  the  eight  with  substantial  or 
extensive  advances  in  production  technology. 


Table  111*3.  Numbers  of  Munitions  for  Which  Selected 
Acquisition  Policies  Were  Applied  at  Each  Required  Level 
of  Technological  Advance 
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Competition  at  the  system  level  during  the  production  phase  was  not  likely  to  be 
used  as  a  means  of  providing  an  alternative  solution  to  requirements  for  advances  in 
production  process  technologies.  Nor  were  warranties  consistently  used  to  ensure  that 
contractors  would  deliver  on  their  performance  and  cost  objectives  as  affected  by  their 
requirements  for  advances  in  technology.  However,  the  only  warranties  used  at  the 
subsystem  level  were  for  munitions  with  extensive  requirements  for  advances  in 
performance,  materials,  and  production  technologies. 

The  data  from  the  questionnaire  provided  conflicting  evidence  of  die  extent  to  which 
anticipated  levels  of  technology  advance  differed  from  actual  levels  of  technology  advance, 
in  terms  of  performance,  materials,  and  production  processes.?  However,  several  of  the 
contractors  indicated  that  the  requirements  for  advances  in  technology  had  been 
underestimated.  A  further  possibility  is  that  difficulties  in  technology  advance  had  been 
anticipated,  but  were  deliberately  understated  in  efforts  to  sell  the  program.  A  more 
detailed  analysis  of  technology  time-of-arrival  is  required  in  order  to  estimate  the 
differences  between  anticipated  and  actual  levels  of  required  technology  advance  and  to 
evaluate  the  risks  associated  with  high  levels  of  technology  advance. 

C .  REQUIREMENTS  FOR  INCREASES  IN  RESOURCES 

Data  were  also  obtained  from  the  questionnaires  on  the  increases  in  three  categories 
of  resources  required  during  the  acquisition  program.  The  three  categories  of  resources 
are:  the  test  equipment  used  during  development  and  production;  production  facilities;  and 
production  tooling.  Increase  in  production  facilities  was  measured  in  terms  of  percentage 
of  square  feet  of  new  floor  area.  Increases  in  production  tooling  and  test  equipment  were 
measured  in  terms  of  percentage  of  dollar  values  of  augmentation  of  existing  tooling  or  test 
equipment,  respectively.  That  there  are  many  problems  in  using  dollar  values  for 
equipment  purchased  at  different  periods  in  time  is  well  recognized.  Because  of  usage  of 
old  equipment,  and  quality  improvements  embodied  in  new  equipment,  calculations  using 
dollar  values  are  not  completely  accurate.  However,  such  calculations  do  give  indications 
of  relative  magnitudes.  Data  on  the  increases  in  these  three  resources  were  again  limited  to 
the  guidance  subsystem. 

The  contractors  submitted  data  showing  both  the  percentage  increases  in  resources 
anticipated  to  be  required  at  the  time  the  development  program  was  initiated,  and  the 
percentage  increases  in  resources  actually  required  during  the  acquisition  program.  The 
data  are  summarized  in  terms  of  the  low,  high,  and  mean  percentage  increases  in  resources 


for  the  munitions  in  the  sample,  so  as  to  be  unidentifiable  as  to  contractor  or  munition. 
These  data  are  shown  in  Figures  ffi-2,  ID-3,  and  III -4,  for  the  percentage  requirements  for 
new  test  equipment,  production  facilities,  and  production  tooling,  respectively.8 
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Notes:  Numbers  in  parentheses  are  sample  sizes.  Bullets  ( *  )  indicate  the  means. 

Figure  111*2.  Range  of  New  Test  Equipment  Requirements 
for  Each  Category  of  Munition 

While  there  appear  to  be  differences  in  the  percentage  requirements  for  new 
resources  based  on  the  categories  of  the  munitions,  only  the  differences  between  new  and 
modified  munitions  are  statistically  significant  The  percentage  requirements  for  new  test 
equipment  and  tooling  were  significantly  higher  for  new  munitions  than  for  modified 
munitions.  Percentage  requirements  for  new  resources  were  not  significantly  affected  by 
when  the  munitions  started  development.9 
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Notes:  Numbers  in  parentheses  are  sample  sizes.  Bullets  ( *  )  indicate  the  means. 

Figure  111-3.  Range  of  New  Facilities  Requirements 
for  Each  Category  of  Munition 

Percentage  requirements  for  new  facilities  and  new  tooling  were  directly  related  to 
the  percentage  requirements  for  new  test  equipment,  but  not  to  each  other.  10  Besides  being 
highly  related  to  each  other,  the  percentage  requirements  for  new  test  equipment  and  new 
tooling  were  both  related  to  the  level  of  technological  advance  required  for  the  production 
processes.  Munitions  that  required  substantial  or  extensive  advances  in  production  process 
technology  also  required  higher  percentages  of  new  test  equipment  and  tooling  than  did 
munitions  that  required  only  minor  advances  in  production  process  technology,  as  shown 
in  Figure  HI-5.11  The  percentage  requirements  for  new  test  equipment  and  tooling  were 
not  significantly  related  to  the  required  levels  of  advance  in  performance  or  materials 
technology.  Nor  were  the  percentage  requirements  for  new  facilities  significantly  related  to 
the  required  levels  of  advance  in  performance,  materials,  or  production  technology. 
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Figure  111*4.  Range  of  New  Tooling  Requirements  for  Each  Category  of  Munition 

•  Munitions  with  higher  percentage  requirements  for  new  test  equipment  and  new 
tooling  were  more  likely  to  have  been  subjected  to  advanced  development  phases, 
independent  cost  estimates,  and  low-rate  initial  production  phases  than  were  munitions  with 
lower  percentage  requirements  for  new  test  equipment  and  tooling,  as  shown  in  Figure 

•  HI-6.12  These  results,  and  the  strong  statistically  significant  relationships  between  the 
percentage  requirements  for  new  test  equipment  and  tooling  and  the  required  levels  of 
advance  in  production  technology  (referred  to  earlier)  support  the  findings  in  the  previous 
section  that  advanced  development  and  independent  cost  estimates  were  more  likely  to  be 

•  applied  to  munitions  with  requirements  for  substantial  or  extensive  advances  in  production 
technology.  The  percentage  requirements  for  new  resources  did  not  vary  significantly  with 
the  applicability  of  other  acquisition  policies,  as  shown  in  Figure  ID-6. 
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Figure  111-5.  Range  of  New  Resource  Requirements 
for  Each  Required  Level  of  Technological  Advance 
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Figure  111-6.  Range  of  New  Resource  Requirements 
for  Applications  of  Each  Acquisition  Policy 
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Figure  MI-6.  Range  of  New  Resource  Requirements  for  Applications 
of  Each  Acquisition  Policy  (Continued) 
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The  quantitative  data  provided  by  the  contractors  show  six  differences  for  four 
separate  munitions,  between  the  anticipated  and  actual  requirements  for  the  three  types  of 
resources.13  Three  of  the  differences  were  small,  for  three  different  munitions.  The  other 
three  differences  were  for  large  underestimates  for  each  of  the  three  resources  for  one 
munition;  that  munition  had  a  much  higher  than  average  development  schedule  growth 
factor  and  production  cost  growth  factor,  but  surprisingly,  it  had  little  development  cost 
growth.  There  were  no  significant  relationships  between  errors  in  the  anticipated 
percentage  requirements  for  new  test  equipment,  new  facilities,  and  new  production 
tooling. 

D.  STABILITY  OF  FUNDING 

A  frequently  heard  complaint  is  that  one  of  the  major  problems  in  acquisition  of 
munitions  (and  other  defense  systems)  is  that  funding  levels  are  not  stable.  Exactly  what 
"stability"  means  in  this  context  is  not  clear.  Year-to-year  variability  in  actual  funding  can 
be  expected  for  a  number  of  reasons.  Development  funding  could  rationally  be  expected  to 
be  lower  in  the  early  phases  of  development  when  concepts  are  being  developed  and 
evaluated  and  the  overall  design  is  being  formulated.  As  more  detailed  design  work 
proceeds,  the  rate  of  development  spending  could  be  expected  to  increase,  and  later  to  taper 
off  as  the  project  nears  production.  Likewise,  production  spending  rates  could  be  expected 
to  be  lower  initially  for  long  lead-time  items  and  low-rate  production.  For  these  reasons, 
some  variation  in  spending  and  funding  rates  should  be  expected. 

What  is  more  likely  meant  by  the  complaint  of  instability  of  funding  is  that  the 
contractor  cannot  reliably  estimate  what  the  funding  will  be  for  future  years.  The  issue  is 
predictability.  At  a  macro  level,  whether  the  overall  defense  appropriation  levels  are 
expected  to  increase  or  decrease  will  likely  affect  the  contractor’s  estimate  of  future  funding 
for  specific  programs.  At  a  more  micro  level,  the  difference  between  the  amount 
appropriated  by  Congress  for  a  specific  munition  in  a  given  year  and  the  corresponding 
amount  submitted  in  the  President's  budget  is  important 

Based  on  these  two  concepts  of  funding  stability,  two  quantitative  measures  were 
calculated.  The  first  measure,  procurement  macro- stability,  consists  of  compound  annual 
growth  rates  in  defense  total  obligational  authority  (TOA)  in  constant  FY  1989  dollars.13 
The  measure  of  procurement  macro-stability  for  each  munition  in  the  sample  was  calculated 
from  the  start  of  production  to  either  the  completion  of  production  or  the  end  of  FY  1989 
(whichever  is  earlier). 


The  second  measure,  procurement  micro-stability,  is  based  on  annual  procurement 
budget  and  appropriation  amounts  shown  in  the  SARs  for  the  munitions  in  the  sample. 
Because  of  differences  in  the  extent  to  which  appropriation  and  budget  data  are  contained  in 
the  SARs,  procurement  micro-measures  could  not  be  calculated  over  all  appropriate  years 
for  all  munitions  in  the  sample,  and  development  micro-measures  could  not  be  calculated 
for  any  of  them.  The  procurement  micro-measure  was  calculated  as  the  root  mean  squared 
annual  percentage  error,  where  the  annual  error  was  defined  as16: 

(the  appropriated  amount  for  the  munition)  -  (the  budgeted  amount  for  the  munition) 

(the  budgeted  amount  for  the  munition) 

The  two  funding  stability  measures  are  shown  in  Table  HI -4  for  each  munition  for 
which  they  could  be  calculated.  The  range  of  macro-measures  is  shown  in  Figure  ID-7  and 
the  range  of  micro-measures  is  shown  in  Figure  HI-8,  for  each  category  of  munition.  I7.1* 
The  only  statistically  significant  difference  in  procurement  TOA  annual  growth  rates  is 
between  Army  and  Air  Force  munitions;  the  two  Air  Force  munitions  entered  procurement 
at  a  time  when  the  procurement  TOA  was  decreasing  at  a  rate  greater  than  experienced  for 
any  of  the  Army  munitions.  The  decreasing  procurement  TOA  for  the  Air  Force  munitions 
was  reflected  in  a  lower  mean  procurement  macro-stability  measure  for  the  Air  Force 
munition  than  for  the  Army  or  Navy  munitions.  The  only  statistically  significant  difference 
in  procurement  micro-stability  measures  was  between  the  Navy  and  Air  Force  munitions. 
There  were  no  statistically  significant  differences  between  Army  and  Navy  munitions,  in 
either  the  procurement  TOA  annual  growth  rates  or  the  procurement  micro  stability 
measure. 

The  procurement  TOA  annual  growth  rates  showed  a  statistically  significant  trend 
over  time.19  They  tended  to  be  lower  for  those  munitions  that  entered  production  later. 
This  simply  reflects  the  large  TOA  growth  rates  in  the  early  1980s,  and  accounts  for  the 
differences  between  Air  Force  and  Army  munitions  alluded  to  above. 

The  root  mean  squared  percentage  difference  between  program  procurement 
requests  and  appropriations  (the  procurement  micro-stability  measure)  did  not  show  any 
trend  over  time.  Nor  were  there  any  statistically  significant  relationships  between  either  of 
the  two  stability  measures  or  with  any  of  the  three  resource  requirement  categories 
discussed  in  the  preceding  section. 20  There  are  no  reasons  to  believe  that  there  should  be 
any  relationships  between  the  procurement  TOA  growth  rates  and  either  the  required  levels 


of  technical  advance  discussed  previously  or  any  of  the  acquisition  policies  discussed  in 
Chapter  n. 


Table  111*4.  Funding  Stability  Measures 


Designator 

Title 

Procurement 

TOA 

Compound 
Annual 
Growth  Rate 

Root  Mean  Squared 
Annual  Percentage 
Difference  Between 
Program  Procurement 
Requests  and  Appropriations 

A/RIM-7E 

Sparrow  ED  B  CW 

- 

AIM-7F 

Sparrow  m  Pulse  Doppler 

6.61 

- 

A/RIM-7M 

Sparrow  HI  Monopulse 

-2.34 

.073 

AIM-9L 

Sidewinder 

8.49 

- 

AIM-9M 

Sidewinder 

2.50 

.247 

AIM-54A 

Phoenix 

-0.84 

— 

AIM-54C 

Phoenix 

4.45 

.122 

AIM-120A 

AMRAAM 

-3.99 

.479 

FIM-92A 

Stinger-Basic 

9.79 

.119 

FIM-92A 

Stinger-POST/RMP 

-2.34 

- 

AGM-65D/F/G 

HR  Maverick 

-3.37 

.508 

A/R/UGM-84  A/C/D  Harpoon 

4.57 

.119 

AGM-88A 

HARM 

2.50 

.276 

AGM-114A/B 

Hellfire 

-0.15 

.092 

BGM-71A 

TOW  I 

0.38 

— 

BGM-71D 

Town 

-0.15 

.236 

- 

MLRS 

4.45 

- 

M-712 

Copperhead  CLGP 

5.91 

.608 

- 

5"  Deadeye  SALOP 

- 

- 

Number  of  Observations 

18 

11 

Average  micro-stability  measures  were  higher,  as  shown  in  Figure  HI-9,  with  but 
one  exception  (no  materials  technology  advance  versus  minor  materials  technology 
advance)  for  higher  levels  of  required  technological  advance  21  This  would  imply  that 
munitions  with  higher  requirements  for  advances  in  technology  were  more  likely  to 
encounter  instability  of  funding.  However,  none  of  these  differences  in  procurement 
micro-stability  between  the  required  levels  of  technological  advance  are  statistically 
significant 

Whether  or  not  certain  of  the  acquisition  policies  were  applied  during  the  production 
phase  did  not  have  any  statistically  significant  effects  on  procurement  micro-stability,  as 
shown  in  Figure  III- 10.22 
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Figure  111-7.  Range  of  Procurement  TOA  Annual  Growth  Rates 
for  Each  Category  of  Munition 
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Figure  1(1-8.  Range  of  Root  Mean  Squared  Annual  Percentage  Differences 
Between  Program  Procurement  Requests  and  Appropriations 
for  Each  Category  of  Munition 
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Figure  111*9.  Range  of  Root  Mean  Squared  Annual  Percentage 
Differences  Between  Program  Procurement  Requests  and  Appropriations, 
for  Each  Required  Level  of  Technological  Advance 
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Figure  111*10.  Range  of  Root  Mean  Squared  Annual  Percentage  Differences 
Between  Program  Procurement  Requests  and  Appropriations, 
for  Applications  of  Each  Acquisition  Policy 
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E.  OVERLAP  BETWEEN  DEVELOPMENT  AND  PRODUCTION 


The  overlap  between  development  and  production  is  defined  as  the  interval  (in 
months)  between  the  startup  of  production  and  the  initial  operation  capability  (IOC)  date. 
The  end  of  the  development  phase  is  commonly  defined  as  the  end  of  development  testing 
and  evaluation,  but  because  of  limitations  in  the  test  data  in  the  SARs,  it  was  not  possible  to 
use  this  definition  for  the  case  studies  in  Volume  II.  Instead,  development  is  here  defined 
as  ending  upon  the  IOC  date.  Production  is  generally  defined  throughout  this  report  as 
starting  with  Milestone  m  or  UIA  (MS  HI),  as  appropriate.  Using  these  definitions  and 
data  from  the  case  studies  in  Volume  H,  development/production  overlap  ratios  were 
calculated  for  twelve  of  the  munitions  in  the  sample  as: 

IOC  -  MS  m 
IOC  -  MS  n 

These  ratios  are  shown  in  Table  HI-5.23  Development/production  overlap  ratios 
were  not  calculated  for  the  MLRS,  which  went  directly  from  advanced  development  to  the 
production  phase,  or  for  the  5"  Deadeye  SALGP,  which  never  entered  production. 
Development/production  overlap  ratios  could  not  be  calculated  for  the  AIM-7E  and  F 
Sparrow,  AIM-54A  Phoenix,  Stinger-Basic,  and  TOW  I,  because  of  insufficient 
information  on  MS  II,  MS  HI,  and  IOC  dates  in  the  SARs. 

Two  caveats  should  be  kept  in  mind  about  the  use  of  these  ratios.  The  first  is  that  a 
value  greater  than  zero  is  entirely  reasonable  and  is  to  be  expected,  because  some 
production  must  take  place  to  provide  the  munitions  for  the  IOC.  Without  some  overlap 
between  development  (as  here  defined)  and  production,  there  can  be  no  IOC.  Because  of 
this,  it  may  not  be  possible  to  say  whether  or  not  the  development/production  overlap  ratio 
for  a  particular  munition  is  excessive.  What  can  be  said  when  development/production 
overlap  ratios  for  different  munitions  are  compared,  is  that  a  higher  ratio  is  worse  than  a 
lower  ratio. 

The  second  caveat  is  that  the  length  of  the  interval  between  the  start  of  the 
production  phase  and  IOC  will  be  the  greater  of:  (1)  any  overlap  or  concurrency  between 
the  start  of  the  production  phase  and  completion  of  testing  and  evaluation;  (2)  the  lead-times 
for  startup  of  production;  and  (3)  any  delays  caused  by  the  availability  of  complementary 
platforms  or  sensors.  For  example,  AIM-9L  Sidewinder  experienced  concurrency  between 
the  start  of  the  production  phase  (April  1976)  and  the  completion  of  testing  (January  1978, 


just  four  months  prior  to  the  IOC  in  May  1978).  Copperhead  experienced  great  difficulties 
in  the  startup  of  production,  as  reflected  in  a  37  month  interval  between  production  startup 
and  IOC.  The  Hellfire's  IOC  was  delayed  significantly  beyond  the  completion  of 
development  testing  due  to  unavailability  of  the  AH-64  Apache.  Because  of  these  two 
caveats,  care  must  be  taken  in  interpreting  differences  in  the  development/production 
overlap  ratios  between  the  munitions  in  the  sample. 


Table  111*5.  Development/Production  Overlap  Ratios 


Designator 

Title 

Development/ 

Production 

Overlap 

Ratio 

A/RIM-7E 

Sparrow  IH-B  CW 

I.U. 

AIM-7F 

Sparrow  in  Pulse  Doppler 

I.U. 

A/RIM-7M 

Sparrow  HI  Monopulse 

.037 

AIM-9L 

Sidewinder 

.346 

AIM-9M 

Sidewinder 

.241 

AIM-54A 

Phoenix 

I.U. 

AIM-54C 

Phoenix 

.689 

AIM-120A 

AMRAAM 

.072 

FIM-92A 

Stinger-Basic 

.371 

FIM-92A 

Stinger-POST/RMP 

.I.U. 

AGM-65D/F/G 

HR  Maverick 

.342 

A/R/UGM-84  A/C/D 

Harpoon 

.625 

AGM-88A 

HARM 

.116 

AGM-114A/B 

Hellfire 

.416 

BGM-71A 

TOW  I 

I.U. 

BGM-71D 

Town 

.410 

- 

MLRS 

N.A. 

M-712 

Copperhead  CLGP 

.529 

— 

5”  Deadeye  SALGP 

N.A. 

N.A.  «  Not  applicable. 


I.U.  =  Information  unavailable. 

There  were  no  statistically  significant  differences  between  development/production 
overlap  ratios  based  on  categories  of  munitions,  as  shown  in  Figure  III- 11.24  Overlaps 
between  development  and  production  did  tend  to  decrease  over  time,  with  lower  ratios  for 
the  munitions  that  entered  production  later. 25  Development/production  overlap  ratios  were 
also  directly  related  to  the  procurement  TOA  annual  growth  rates,  which  should  be 
expected  because  of  the  strong  inverse  relationship  between  production  start  dates  and 
procurement  TOA  annual  growth  rates  discussed  in  the  preceding  section.  However,  there 
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was  no  statistically  significant  relationship  between  the  development/production  overlap 
ratios  and  the  procurement  micro-stability  measures. 

The  development/production  overlap  ratios  did  not  vary  significantly  with  either  the 
percentage  requirements  for  new  resources,  or  with  the  requirements  for  advances  in 
technology  (as  shown  in  Figure  III-12).26.  27  However,  munitions  with  system 
production  competition  did  tend  to  have  lower  development/production  overlap  ratios  than 
did  munitions  without,  and  those  differences  were  statistically  significant,  as  shown  in 
Figure  HI- 13.28 
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Notes:  Numbers  in  parentheses  are  sample  sizes.  Bullets  (•  )  indicate  the  means. 

Figure  111-11.  Range  of  Development/Production  Overlap  Ratios 
for  Each  Category  of  Munition 

F.  AVAILABILITY  OF  REQUIRED  COMPLEMENTARY  SYSTEMS 

The  development  schedules  for  two  of  the  nineteen  munitions  in  the  sample  were 
significantly  affected  by  the  availability  of  required  complementary  systems.  The 
AMRAAM  planned  development  schedule  was  compressed  in  order  to  provide  missiles  for 
use  by  F-16  aircraft  in  the  NATO  theater.  The  IOC  of  the  Hellfire  was  delayed  from  FY 
1985  until  July  1986  because  of  delays  in  development  and  operational  schedule  for  the 
AH-64  helicopter.  In  both  cases,  the  complementary  system  was  the  platform.  For  several 
of  the  other  munitions  in  the  sample,  the  use  of  special  target  designators  was  required,  but 


the  development  schedules  for  those  munitions  were  not  adversely  affected  by  the 
availability  of  the  designators. 


Minor  Performance  Advances  (1) 
Substantial  Performance  Advances  (3) 
Extensive  Performance  Advances (4) 

No  Materials  Advances  (1) 

Minor  Materials  Advances  (1 ) 
Substantial  Materials  Advances  (3) 
Externsive  Materials  Advances  (3) 

Minor  Production  Advances  (2) 
Substantial  Production  Advances  (4) 
Extensive  Production  Advances  (2) 


Notes:  Numbers  in  parentheses  are  sample  sizes.  Bullets  (  •  )  indicate  the  means. 


Figure  111-12.  Range  of  Development/Production  Overlap  Ratios 
for  Each  Required  Level  of  Technological  Advance 
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With  Independent  Cost  Estimate  (5) 
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Without  Low-Rate  Initial  Production  Phase  (3) 

With  System  Production  Competition  (6) 
Without  System  Production  CompetKion  (6) 
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Notes:  Numbers  In  parentheses  are  sample  sizes.  Bullets  ( •  )  Indicate  the  means. 


Figure  111-13.  Range  of  Development/Production  Overlap  Ratios 
for  Applications  of  Each  Acquisition  Policy 


This  may  not  always  be  the  case  in  the  future.  The  lesson  is  that  the  progress  of  a 
munition's  acquisition  program  should  not  be  managed  in  isolation;  consideration  should 
be  given  to  the  progress  of  the  acquisition  programs  for  complementary  systems  such  as 
platform  and  target  designators.  In  the  Hellfire  example,  the  actual  engineering 
development  program  could  have  been  increased  by  at  least  9  months  without  adversely 
affecting  the  July  1986  IOC  of  the  AH-64  armed  with  that  missile.  It  would  have  been  a 
waste  to  expend  funds  for  overtime  during  the  development  of  the  Hellfire. 


IV.  COMPARISONS  OF  PROGRAM  OUTCOMES 


In  this  chapter,  the  nineteen  munitions  are  compared  in  terms  of  several  measures  of 
program  success,  and  are  evaluated  in  terms  of  how  those  measures  were  related  to  the 
munition  characteristics,  the  acquisition  policies  that  were  applied,  and  the  risks  that  were 
encountered.  The  measures  of  program  success  used  are: 

•  Satisfaction  of  operational  and  technical  requirements; 

•  Development  schedule  growth; 

•  Development  quantity  growth; 

•  Development  cost  growth; 

•  Production  cost  growth; 

•  Production  quantity  growth; 

•  Production  stretchout 

•  Total  program  cost  growth. 

These  measures  were  selected  because  data  for  each  could  be  obtained  from  the 
Selected  Acquisition  Reports  (SARs).  For  other  measures  of  program  success  that  could 
also  be  defined,  the  data  were  not  readily  available. 

A.  SATISFACTION  OF  OPERATIONAL  AND  TECHNICAL 
REQUIREMENTS 

Satisfaction  of  operational  and  technical  requirements  is  defined  in  the  same  terms 
as  used  in  the  SARs.  Operational  requirements  include  measures  of  reliability  and 
maintainability.  Technical  requirements  include  measures  of  size  and  weight,  although 
weight  is  included  under  operational  requirements  in  at  least  one  of  the  SARs  (AIM-7F 
Sparrow).  Performance  requirements  such  as  range,  velocity,  and  accuracy  are  included 
under  either  operational  requirements  or  technical  requirements  in  the  SARs,  depending 
upon  the  particular  munition.  Because  of  the  somewhat  arbitrary  assignment  of  these 
requirements  in  the  SARs,  the  two  types  of  requirements  have  been  combined  in  the 
following  paragraphs. 

The  data  showing  whether  or  not  the  operational  and  technical  requirements  listed  in 
the  SARs  were  totally  satisfied  are  shown  in  Table  IV- 1.  Two  of  the  munitions  in  the 


sample  have  not  completed  development  testing:  the  AMRAAM  and  the  Stinger- 
POST/RMP.  Although  development  testing  has  not  been  completed  for  the  Stinger- 
POST/RMP,  testing  to  date  has  shown  deficient  reliability. 

Table  IV-1.  Satisfaction  of  Operational  and  Technical  Requirements 


Designator 

Title 

New/ 

Mod 

Requirements  Satisfied 

A/RiM-7E 

Sparrow  HI-3  CW 

Mod 

Yes 

AIM-7F 

Sparrow  HI  Pulse  Doppler 

Mod 

Overweight,  deficient  speed,  range  and  altitude 

A/RIM-7M 

Sparrow  HI  Monopulse 

Mod 

Deficient  reliability  and  guidance  probability 

AIM-9L 

Sidewinder 

Mod 

Deficient  head-on  range 

AIM-9M 

Sidewinder 

Mod 

Deficient  head-on  range  and  flare  discrimination 

AIM-54A 

Phoenix 

New 

Launch  weight  5  percent  over 

AIM-54C 

Phoenix 

Mod 

Yes 

AIM-120A 

AMRAAM 

New 

Unknown-development  testing  not  yet  completed 

FIM-92A 

Stinger-Basic 

New 

Yes 

FIM-92A 

Stinger-POST/RMP 

Mod 

Unknown-development  testing  not  yet  completed 

AGM-65D/F/G 

HR  Maverick 

Mod 

Yes 

A/R/UGM-84  A/C/D 

Harpoon 

New 

Launch  weight  5  percent  over 

AGM-88A 

HARM 

New 

Deficient  classified  requirement 

AGM-114A/B 

Hellfire 

New 

5  percent  overweight 

BGM-71A 

TOW  I 

New 

Slightly  overweight 

BGM-71D 

Town 

Mod 

Yes 

- 

MLRS 

M-712 

Copperhead  CLGP 

New 

Yes 

5"  Deadeye  SALGP 

New 

Yes 

For  ten  of  the  seventeen  munitions  for  which  development  testing  had  been 
completed,  the  requirements  listed  in  the  SARs  were  not  totally  satisfied.  Head-on-range 
and  flare  discrimination  requirements  were  not  completely  satisfied  for  the  AIM-9L  and 
AIM-9M  Sidewinder  missiles,  but  the  attained  performance  of  these  missiles  is  militarily 
useful.  The  Harpoon,  the  Hellfire,  and  the  TOW  I  are  all  slightly  overweight,  but  not 
enough  to  require  additional  development  effort  to  reduce  the  weight  The  accuracy  of  the 
MLRS  is  below  specifications,  but  again,  the  demonstrated  accuracy  is  militarily  useful, 
and  the  additional  cost  that  would  be  required  to  achieve  the  specified  accuracy  was  judged 
to  be  excessive.  For  the  one  munition  in  the  sample  that  was  cancelled  prior  to  production 
(5"  Deadeye  SALGP),  both  the  operational  and  technical  requirements  listed  in  the  SAR 
were  satisfied. 


Satisfaction  of  the  operational  and  technical  requirements  listed  in  the  SARs  for 
each  category  of  munition  is  shown  in  Table  IV- 2.  Satisfaction  of  operational  and  technical 
requirements  did  not  differ  significantly  between  the  various  categories  of  munitions.1 

Table  IV-2.  Differences  Between  Munition  Categories 
and  Satisfaction  of  Operational  and  Technical  Requirements 


Requirements 

Satisfied 

Requirements 

Not  Satisfied 

Total 

Total  Sample 

7 

10 

17 

Intercept 

3 

5 

8 

Surface  Attack 

4 

5 

9 

Air  Launch 

3 

8 

11 

Ground  Launch 

4 

2 

6 

Army 

3 

3 

6 

Navy 

3 

7 

10 

Air  Force 

1 

0 

1 

New 

3 

6 

9 

Modification 

4 

4 

8 

Of  the  seven  munitions  whose  requirements  were  completely  satisfied,  85.7  percent 
(=6/7)  were  deployed  and  14.3  percent  (=1/7)  were  never  deployed.  Of  the  sixteen 
munitions  that  have  been  deployed,  37.5  percent  (=  6/16)  completely  satisfied  the 
operational  and  technical  requirements  listed  in  the  SARs,  and  62.5  percent  (=  10/16)  did 
not  fully  satisfy  those  requirements.  These  results  suggest  that: 

•  A  significant  level  of  uncertainty  exists  about  the  relationship  between 
deployment  and  satisfaction  of  the  operational  and  technical  requirements. 

•  There  was  some  margin  for  design  tradeoffs. 

•  Operational  and  technical  requirements  for  a  substantial  share  of  these 
munitions  may  have  been  unrealistically  stringent 

Whether  or  not  the  operational  and  technical  requirements  listed  in  the  SARs  were 
satisfied  is  not  related  to  when  the  munition  entered  development.2  Nor  do  any  of  the 
acquisition  policies  appear  to  have  significantly  increased  the  proportions  of  munitions 
satisfying  those  requirements,  as  shown  in  Table  IV-3.3 

Whether  or  not  the  requirements  listed  in  the  SARs  were  satisfied  was  not 
significantly  related  to  the  levels  of  technological  advance  that  were  required.  As  shown  in 
Table  IV-4,  the  proportion  of  munitions  satisfying  the  operational  and  technical 
requirements  was  lower  for  munitions  with  extensive  requirements  for  advances  in 


performance  or  production  technology  than  for  munitions  with  lesser  requirements  for 
technological  advance,  but  the  differences  are  not  statistically  significant4 

Table  IV-3.  Numbers  of  Munitions  Satisfying  Operational 
and  Technical  Requirements  for  Applications 
of  Each  Acquisition  Policy 

Policy  Applied 


Yes 

No 

Advanced  Development 

4/11 

3/6 

Competitive  Advanced  Development 

1/6 

3/5 

Advanced  Development  Prototype 

2/9 

4/7 

Full  Scale  Development  Subsystem  Compedtion 

1/4 

6/12 

Independent  Testing 

1/3 

5/13 

Independent  Cost  Estimate 

3/5 

3/10 

Low-Rate  Initial  Production 

3/8 

3/6 

Table  1V-4.  Numbers  of  Munitions  Satisfying  Operational 
and  Technical  Requirements  for  Required  Levels 
of  Technological  Advance 

Requirements  for  Technology  Advance 


Off-the-Shelf 

Minor 

Substantial 

Extensive 

Total 

Performance  Advances 

Satisfied 

0 

0 

4 

2 

6 

Not  Satisfied 

0 

1 

0 

2 

3 

Total 

0 

1 

4 

4 

9 

Materials  Advances 

Satisfied 

1 

0 

3 

2 

6 

Not  Satisfied 

0 

1 

1 

1 

3 

Total 

1 

1 

4 

3 

9 

Production  Advances 

Satisfied 

0 

2 

3 

1 

6 

Not  Satisfied 

0 

0 

2 

1 

3 

Total 

0 

2 

5 

2 

9 

The  ranges  of  percentage  requirements  for  new  resources  were  lower  for  the 
munitions  satisfying  the  requirements  than  for  the  munitions  that  did  not,  as  shown  in 
Figure  IV- 1.5  However,  only  the  difference  in  test  equipment  requirements  between  the 
two  groups  of  munitions  was  statistically  significant.  Each  of  the  munitions  that  did  not 
fully  satisfy  the  requirements  needed  completely  new  test  equipment. 
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Notes:  Numbers  in  parentheses  are  sample  sizes.  Bullets  (  • )  indicate  the  means. 


Figure  IV-1.  Range  of  New  Resource  Requirements  for  Whether  or  Not 
•  Operational  and  Technical  Requirements  Were  Satisfied 

Differences  in  development/production  overlap  ratios,  depending  on  whether  or  not 
the  munitions  satisfied  the  operational  and  technical  requirements,  as  shown  in  Figure  IV- 
2,  were  not  statistically  significant^ 


Requirements  Satisfied  (5) 

•  Requirements  Not  Satisfied  (6) 

100% 

Notes:  Numbers  in  parentheses  are  sample  sizes.  Bullets  (  • )  indicate  the  means. 

Figure  IV-2.  Range  of  Development/Production  Overlap  Ratios  for  Whether 
or  Not  Operational  and  Technical  Requirements  Were  Satisfied 

Of  the  two  munitions  in  the  sample  whose  performance  requirements  were 
changed  while  the  munitions  were  still  in  the  development  phase,  one  (ITR  Maverick) 
satisfied  its  requirements,  and  the  other  (HARM)  did  not.  The  one  munition  in  the 
sample  whose  IOC  was  delayed  by  the  unavailability  of  its  major  platform  (Hellfire, 
delayed  by  the  development  of  the  AH-64)  did  not  fully  satisfy  its  operational  and  technical 
requirements. 


IV-5 


B  .  DEVELOPMENT  SCHEDULE  GROWTH 


The  durations  of  advanced  development  and  full  scale  development,  and  the  full 
scale  development  schedule  growth  factor,  are  shown  in  Table  IV-5  for  each  of  the 
munitions  in  the  sample.  Advanced  development  is  defined  as  .the  number  of  months  from 
Milestone  I  or  the  awarding  of  the  advanced  development  contract  (whichever  is  later),  to 
Milestone  II  or  the  awarding  of  the  full  scale  development  (FSD)  contract  (whichever  is 
earlier).  FSD  is  defined  as  ending  upon  the  IOC  date.  The  development  schedule  growth 
factor  is  defined  as  the  number  of  months  from  Milestone  II  to  the  actual  IOC  date,  divided 
by  the  number  of  months  from  Milestone  II  to  the  IOC  date  planned  at  Milestone  n.  For 
those  munitions  for  which  there  were  no  Milestone  II  dates  in  the  SARs,  the  growth  factor 
was  based  on  the  start  of  FSD. 

As  stated  in  Chapter  II,  only  two  of  the  modified  munitions  in  the  sample  had 
advanced  development  (AIM-7M  Sparrow  and  IIR  Maverick),  but  all  of  the  new  munitions 
did.  However,  information  as  to  when  three  of  the  munitions  (Stinger-Basic,  HR 
Maverick,  and  5"  Deadeye  SALGP)  entered  and  completed  advanced  development  could 
not  be  obtained. 

The  range  of  months  in  advanced  development  for  the  nine  munitions  for  which 
dates  are  available,  and  the  ranges  of  planned  and  actual  months  in  FSD  for  each  category 
of  munition  are  shown  in  Figure  IV-3.7  Advanced  development  required  over  three  years, 
on  the  average.  FSD  required  over  seven  years,  on  the  average.  FSD  for  modified 
munitions  actually  took  almost  a  year  longer  on  the  average  than  for  new  munitions. 

The  actual  months  in  FSD  increased  by  almost  half,  on  the  average,  over  the 
planned  duration.  As  shown  in  Figure  IV-4,  development  schedule  growth  factors  did  not 
vary  greatly  between  munition  categories,  except  for  the  significantly  higher  development 
growth  factors  for  the  two  Air  Force  munitions  as  compared  to  the  seven  Army  munitions. 8 
Development  schedule  growth  was  at  least  as  bad  for  modified  munitions  as  for  new 
munitions;  there  was  no  statistically  significant  difference  between  these  two  groups. 

There  were  no  statistically  significant  differences  in  development  schedule  growth 
factors  depending  on  whether  or  not  the  acquisition  policies  considered  in  this  report  were 
applied,  as  shown  in  Figure  IV-5 .9  In  particular,  advanced  development,  prototyping,  and 
competitive  subsystem  development  did  not  result  in  significantly  higher  development 
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Notes:  Numbers  In  parentheses  are  sample  sizes.  Bullets  (•  )  indicate  Ihe  means. 

Figure  IV-S.  Range  of  Development  Schedule  Growth  Factors 
for  Applications  of  Each  Acquisition  Policy 
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schedule  growth.  Furthermore,  there  were  no  statistically  significant  differences  in 
development  schedule  growth  factors  related  to  the  risks  described  in  Chapter  m,  as  shown 
in  Figure  IV-6.10 


Stable  Threat  (17) 

Unstable  Threat  (2) 

Technology  Advance  Subsample  (1 1) 
Remainder  of  Sample  (8) 

Minor  Performance  Advance  (1) 
Substantial  Performance  Advance  (4) 
Extensive  Performance  Advance  (6) 
No  Materials  Advance  (1) 

Minor  Materials  Advance  ( 1 ) 
Substantial  Materials  Advance  (5) 
Extensive  Materials  Advance  (4) 
Minor  Production  Advance  (2) 
Substantial  Production  Advance  (6) 
Extensive  Production  Advance  (3) 

Development/Production 
Overlap  Sub6ample  ( 1 2) 

Remainder  of  Sample  (7) 

1.0  1.5  2.0  2.5  3.0 

Notes:  Numbers  in  parentheses  are  sample  sizes.  Bullets  ( •  )  indicate  the  means. 

Figure  IV-6.  Range  of  Development  Schedule  Growth  Factors 
for  Each  Category  of  Program  Risk 


Development  schedule  growth  factors  for  the  two  munitions  which  had  substantial 
threat  changes  did  not  differ  from  munitions  for  which  the  threats  were  stable.  Munitions 
with  extensive  requirements  for  advances  in  technology  did  not  suffer  any  significantly 
greater  slippage  in  development  schedules  than  did  munitions  with  lesser  requirements  for 
advances  in  technology. 

Requirements  for  new  test  equipment,  facilities,  or  tooling  did  not  affect  the 
completion  of  the  development  schedule.  H  Munitions  with  low  percentage  requirements 
for  new  resources  were  no  more  likely  to  complete  development  on  schedule  than  were 
munitions  with  high  percentage  requirements  for  new  resources. 

Development  schedule  growth  factors  did  not  show  any  significant  changes  over 
time.  Nor  were  development  schedule  growth  factors  correlated  with  the 
development/production  overlap  ratios  discussed  in  Chapter  EL  12 

The  numbers  of  months  in  advanced  development  had  no  statistically  significant 
relationships  with  development  schedule  growth  factors  or  the  numbers  of  months  in  full 
scale  development.  The  absence  of  any  correlation  is  in  agreement  with  the  absence  of 
any  statistically  significant  differences  in  development  schedule  growth  factors  between 
munitions  with  and  munitions  without  advanced  development  phases,  as  shown  in 
Figure  IV-5. 

C .  DEVELOPMENT  QUANTITY  GROWTH 

The  numbers  of  test  articles  (both  the  development  estimates  and  the  current 
estimates)  and  the  development  quantity  growth  factors,  are  shown  in  Table  IV-6.  The 
development  quantity  growth  factor  is  defined  as  the  current  estimate  of  the  number  of  test 
articles  shown  in  the  latest  SAR  divided  by  the  estimate  shown  in  the  SAR  at  the  time  of 
Milestone  II  approval.  For  those  munitions  for  which  Milestone  II  dates  do  not  appear  in 
the  SAR,  the  growth  factors  were  based  on  estimates  as  of  the  start  of  FSD  from  the 
earliest  available  SAR. 

Two  of  the  munitions  in  the  sample  have  not  completed  development:  the 
AMRAAM  and  the  Stinger-POST/RMP.  The  first  is  a  new  munition  and  the  second  a 
modification.  The  current  estimate  of  the  number  of  test  articles  for  the  AMRAAM  has 
been  reduced  below  the  development  estimate  by  a  third  in  order  to  limit  the  increased 
development  costs  experienced  by  that  munition.  The  current  estimate  for  the  Stinger- 


Table  IV-6.  Development  Quantities  and  Development  Quantity  Growth  Factors 
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POST/RMP  shows  no  increase  in  the  number  of  test  articles  over  the  development  estimate. 
However,  it  is  possible  that  the  number  of  test  articles  for  both  the  AMRAAM  and  the 
Stinger-POST/RMP  could  change  before  development  is  completed. 

The  ranges  of  development  ("planned”)  and  current  estimates  of  test  article 
quantities  for  each  category  of  munition  are  shown  in  Figure  IV-7.H  The  patterns  in  the 
figure  result  primarily  from  two  causes.  The  first  cause  is  the  large  development  estimates 
of  test  article  quantities,  and  somewhat  smaller  (although  still  large)  current  estimates  of  test 
article  quantities,  for  new  Army  surface  attack,  surface-launched  munitions.  The  average 
numbers  of  test  articles  for  these  categories  of  munitions  are  more  than  double  the  average 
number  of  test  articles  for  the  corresponding  categories.  The  second  cause  is  that  the 
current  estimates  of  test  article  quantities  for  modifications  of  Navy  air-launched  intercept 
munitions  were  in  all  cases  as  high  or  much  higher  than  the  development  estimates  because 
of  optimistic  initial  assessments  of  the  technical  difficulties.  This  caused  the  average 
current  estimates  of  test  article  quantities  to  be  higher  than  the  development  estimates  for  the 
intercept,  air-launched.  Navy,  and  modified  munitions  categories. 

These  patterns  are  reflected  in  the  ranges  of  development  quantity  growth  factors 
for  each  category  of  munition,  as  shown  in  Figure  IV-8.15  The  development  quantity 
growth  factors  are  significantly  greater  for  air-launched  and  modified  munitions  than  for 
surface-launched  and  new  munitions,  respectively.  While  the  ranges  of  development 
quantity  growth  factors  are  higher  for  intercept  and  Navy  munitions  as  compared  to  surface 
attack,  and  Army  and  Air  Force  munitions,  respectively,  those  differences  are  not 
statistically  significant. 

Technical  problems  resulting  in  development  schedule  growth  did  not  necessarily 
result  in  development  quantity  growth.  Development  quantity  growth  factors  were  not 
significantly  correlated  with  development  schedule  growth  factors,  even  when  the  sample  is 
divided  into  new  and  modified  munitions.  16  However,  development  quantity  growth  was 
lower  for  the  more  recently  developed  munitions. 

The  extent  of  development  quantity  growth  was  significantly  related  to  the 
applicability  of  several  of  the  acquisition  policies,  as  shown  in  Figure  IV-9.17  The  range  of 
development  quantity  growth  factors  was  lower  for  munitions  with  independent  cost 
estimates  than  for  those  without.  The  range  of  development  quantity  growth  factors  was 
also  significantly  lower  for  munitions  with  advanced  development  phases,  whether  with  or 
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Figure  IV-8.  Range  of  Development  Quantity  Growth  Factors 
for  Each  Category  of  Munition 
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IV-9.  Range  of  Development  Quantity  Growth 
for  Applications  of  Each  Acquisition  Policy 
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without  prototypes.  This  result  is  closely  related  to  the  differences  in  development  quantity 
growth  factors  between  new  and  modified  munitions,  in  that  all  of  the  new  munitions  and 
only  two  of  the  modified  munitions  had  advanced  development  phases.  Optimism  about 
technical  problems  in  the  development  of  modifications  may  have  resulted  in  the  foregoing 
of  advanced  development  phases  as  well  as  underestimates  of  the  numbers  of  test  articles 
that  would  be  required.  Development  quantity  growth  for  the  modified  munitions  with 
advanced  development  was  as  low  or  lower  than  for  any  of  the  modified  munitions  without 
advanced  development,  but  the  subsample  size  for  modifications  was  too  small  to  show 
that  the  differences  are  statistically  significant 

The  extent  to  which  development  quantity  growth  was  related  to  the  risks  described 
in  Chapter  in  is  shown  in  Figure  IV- 10. 18  The  two  munitions  that  had  substantial  changes 
to  the  threat  during  the  development  phase  either  had  no  growth  (HARM)  or  a  slight 
decrease  in  the  numbers  of  test  articles  (HR  Maverick).  Changes  in  the  threat  did  not  result 
in  any  increase  in  the  number  of  test  articles  required. 

As  shown  in  Figure  IV-10,  the  development  quantity  growth  factors  for  the 
technology  advance  and  resource  requirements  subsample  discussed  in  Chapter  HI  are  not 
representative  of  the  total  sample.  Because  of  the  differences  between  this  subsample  and 
the  remainder  of  the  sample,  no  valid  inferences  could  be  drawn  for  the  entire  sample  about 
relationships  between  development  quantity  growth  and  the  required  levels  of  technological 
advance  or  percentage  requirements  for  new  resources. 

Development  quantity  growth  was  not  correlated  with  the  development/production 
overlap  ratios  discussed  in  the  previous  chapter.  19 


The  number  of  test  articles  was  inversely  related  to  the  average  cost  of  the  test 
articles.  Log-linear  relationships  for  both  new  and  modified  munitions  are  shown  in 
Figure  IV- 1 1.20  The  more  complex  and  expensive  new  munitions  had  smaller  numbers  of  • 

test  articles,  while  less  expensive  munitions  such  as  TOW  and  MLRS  had  relatively 
larger  numbers  of  test  articles.  The  relationship  does  not  appear  to  be  as  strong  for 
modification  programs.  What  is  not  apparent  is  whether  or  not  more  simulation  work  was 
done  with  the  more  expensive  munitions  in  order  to  compensate  for  the  fewer  numbers  of  • 

test  articles. 
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■  New  programs 


Figure  IV-11.  Cost  Per  Test  Article 


D.  DEVELOPMENT  COST  GROWTH 


The  development  and  current  estimates  of  development  costs,  and  the  development 
cost  growth  factors  are  shown  in  Table  IV-7.  All  of  the  cost  estimates  have  been  escalated 
to  1989  dollars.21  The  development  cost  growth  factor  is  defined  as  the  current  estimate 
shown  in  the  latest  SAR  divided  by  the  development  estimate  shown  in  the  SAR  at  the  time 
of  Milestone  II  approval.  For  those  munitions  for  which  no  Milestone  II  dates  appear  in 
the  SARs,  growth  factors  were  based  on  the  start  of  full  scale  development 

Two  of  the  munitions  in  the  sample  have  not  completed  development:  the 
AMRAAM  and  the  Stinger-POST/RMP.  The  development  cost  growth  factors  for  those 
two  munitions  will  continue  to  increase,  above  the  latest  values  of  1.50  and  1.02, 
respectively,  until  development  is  completed.  For  the  remainder  of  the  munitions  in  the 
sample,  the  current  estimates  are  the  actual  development  costs. 

The  ranges  of  development  and  current  estimates  for  each  category  of  munition  are 
shown  in  Figure  IV-12.22  For  each  category,  the  average  of  the  current  estimates  is  higher 
than  the  average  of  the  development  estimates.  While  the  average  for  the  Air  Force 
munitions  is  more  than  twice  as  high  as  for  the  Army  and  Navy  munitions,  this  is  due  to 
the  highest  development  cost  munition  (the  AMRAAM)  being  included  in  a  sample  size  of 
two. . 

The  average  development  cost  for  new  munitions  was  over  twice  that  for  modified 
munitions,  and  little  overlap  was  apparent  between  the  ranges  of  development  costs.  Even 
with  substantial  cost  growth,  the  development  costs  for  modifications  tended  to  be  much 
lower  than  for  new  munitions. 

Ranges  of  development  cost  growth  factors  for  each  category  of  munition  are 
shown  in  Figure  IV- 13.23  Cost  growth  for  intercept  munitions  was  significantly  higher 
than  for  surface  attack  munitions,  primarily  due  to  the  AIM-7F  Sparrow  and  both  of  the 
AIM-9  Sidewinder  modifications.  For  the  same  reason,  the  average  development  cost 
growth  was  higher  for  the  modified  and  air-launched  munitions  categories,  although  the 
differences  between  those  and  the  new  and  surface-launched  munition  categories, 
respectively,  were  not  statistically  significant. 

It  should  not  be  surprising,  then,  that  development  cost  growth  and  development 
quantity  growth  were  directly  correlated  for  the  total  sample  and  for  the  air-launched  and 


Table  IV-7.  Development  Costs  and  Development  Cost  Growth  Factors 
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Figure  IV-12.  Ranges  of  Planned  and  Current  Estimates 
of  Development  Costs  for  Each  Category  of  Munition 


Figure  IV-13.  Range  of  Development  Cost  Growth  Factors 
for  Each  Category  of  Munition 


modified  categories.24  These  correlations  largely  result  from  the  high  development  cost 
growth  and  high  development  quantity  growth  for  four  of  the  six  Navy  air-launched 
intercept  missile  modifications  (AIM-7F  Sparrow,  AIM-9L  and  AIM-9M  Sidewinder,  and 
AIM-54C  Phoenix) . 

Surprisingly,  development  cost  growth  was  not  related  to  development  schedule 
growth,  for  the  entire  sample,  new  munitions,  or  modified  munitions.  Nor  was  any 
relationship  detected  between  development  cost  growth  and  full  scale  development  start 
date.25 

The  extent  of  development  cost  growth  was  significantly  related  to  the  application 
of  several  of  the  acquisition  policies,  as  shown  in  Figure  IV- 14.26  As  was  the  case  with 
development  quantity  growth,  development  cost  growth  was  significantly  lower  for 
munitions  with  advanced  development  phases.  Again,  this  result  is  closely  related  to  the 
differences  in  development  cost  growth  between  new  and  modified  munitions,  in  that  all  of 
the  new  munitions,  but  only  two  of  the  modified  munitions  underwent  advanced 
development  The  average  of  the  development  cost  growth  factors  for  the  modified 
munitions  with  advanced  development  phases  was  less  than  half  of  the  average  of  the 
development  cost  growth  factors  for  the  modified  munitions  without  advanced  development 
phases;  however,  the  sample  size  for  modifications  was  too  small  to  show  that  the 
differences  are  statistically  significant.  Whether  or  not  advanced  development  was 
competitive  made  no  difference,  but  development  cost  growth  was  significantly  lower  for 
munitions  with  competitive  full  scale  development  at  the  subsystem  level. 

The  extent  to  which  development  cost  growth  was  related  to  the  risks  described  in 
Chapter  HI  is  shown  in  Figure  IV- 15.27  For  the  two  munitions  that  had  substantial 
changes  to  the  threat  during  the  development  phase  (HR  Maverick  and  HARM),  the  effects 
of  threat  instability  on  development  cost  growth  were  ambiguous.  The  HR  Maverick  had 
the  third  lowest  development  cost  growth  among  both  the  nine  surface  attack  munitions  and 
the  nine  modifications,  and  the  fourth  lowest  development  cost  growth  among  the  twelve 
air-launched  munitions.  The  HARM  had  the  second  highest  development  cost  growth 
among  the  nine  surface  attack  munitions,  the  sixth  highest  among  the  twelve  air-launched 
munitions,  and  the  third  highest  among  the  ten  new  munitions.  The  difference  in 
development  cost  growth  between  these  two  munitions  suggests  that  changes  to  the  threat 
need  not  always  result  in  high  development  cost  growth. 
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IV-14.  Range  of  Development  Cost  Growth  Factors 
for  Applications  of  Each  Acquisition  Policy 
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Figure  IV-15.  Range  of  Development  Cost  Growth  Factors 
for  Each  Category  of  Program  Risk 

Development  cost  growth  was  not  related  to  the  required  levels  of  technological 
advance  for  performance,  materials,  or  production  processes,  as  shown  in  Figure  IV-15. 
Higher  technological  risks  do  not  necessarily  result  in  higher  development  cost  growth. 
Development  cost  growth  was  inversely  correlated  with  the  percentage  requirements  for 
new  test  equipment,  facilities,  and  tooling,  but  only  the  correlation  with  facilities  is 
statistically  significant.28  Higher  development  cost  growth  may  result  in  lower 
expenditures  on  facilities. 


Development  cost  growth  was  not  affected  by  the  extent  of  overlap  between 
development  and  production.29 

E.  PRODUCTION  COST  GROWTH 

The  estimates  of  production  costs  made  at  the  start  of  development,  the  current 
estimates  of  production  costs,  what  the  production  costs  would  be  if  the  development 
estimate  quantity  were  produced,  and  the  production  cost  growth  factors  are  shown  in 
Table  IV-8.30  The  development  estimates  of  production  costs  are  from  the  earliest  available 
SARs  for  the  munitions,  subsequently  escalated  from  then-year  dollars  to  1989  dollars. 
The  current  estimates  are  from  the  latest  SARs,  also  escalated  to  1989  dollars.  The 
production  cost  growth  factors  are  based  on  estimates  of  what  the  production  costs  would 
be  if  the  quantity  forecast  at  the  time  of  Milestone  II  were  produced.  Those  estimates, 
taken  from  the  individual  munition  case  studies  in  Volume  n,  are  based  on  calculations  of 
total  procurement  (recurring  and  non-recurring)  cost-quantity  curve  parameters,  which  are 
then  used  to  estimate  the  quantity  adjusted  production  costs.  Because  of  insufficient  data,  it 
was  not  possible  to  do  those  calculations  for  the  A/RIM-7E  and  AIM-7F  Sparrow 
modifications,  the  AMRAAM,  the  Stinger-POST/RMP,  the  TOW  I,  or  the  MLRS. 
Production  cost  growth  factors  calculated  in  this  manner  were  used  in  the  comparisons  in 
order  to  avoid  the  distortions  to  total  production  costs  caused  by  changes  in  total  production 
quantities.  Total  production  quantity  changes  are  discussed  in  the  next  section  of  this 
chapter. 

Ranges  of  the  estimates  of  production  costs  made  at  the  start  of  development 
("planned")  and  what  the  current  production  costs  would  be  if  the  development  estimate 
quantity  were  produced  ("adjusted")  are  shown  in  Figure  IV- 16  for  each  category  of 
munition.3i  For  each  category,  the  average  of  the  current  total  estimates  is  higher  than  the 
average  of  the  originally  planned  estimates.  The  average  for  the  surface  attack  category  is 
almost  twice  as  high  as  the  average  for  the  intercept  category.  The  current  average  for 
modifications  is  approximately  two-thirds  of  the  current  average  for  new  munitions. 

Ranges  of  the  production  cost  growth  factors  for  each  category  of  munition  are 
shown  in  Figure  IV- 17.32  Production  cost  growth  does  not  vary  significantly  between  any 
of  the  categories.  While  production  cost  growth  tended  to  be  somewhat  higher  for  new 
munitions  than  for  modifications,  the  difference  is  not  statistically  significant. 


Table  IV-8.  Development,  Current,  and  Quantity-Adjusted  Estimates 
of  Production  Costs,  and  Production  Cost  Growth  Factors 


Designator 

Title 

New/ 

Mod 

Development 
pflimatf  of 
Production 
Cost 

(1989$  Mi  lions) 

Current 

of  Production 
Cost 

(1989$  Millions) 

Quantity 
Adjuaed 
Estimate  of 
Production  Coat 
(1989S  Millions) 

Production 
Coat  Growth 
Factor 

A/RIM-7E 

Sparrow  HI  B  CW 

Mod 

4,9223 

2,126.8 

LD. 

LD. 

AIM-7F 

Sparrow  in  Pulse  Doppler 

Mod 

1.498.9 

3,645.8 

I.D. 

LD. 

A/RIM-7M 

Sparrow  IQ  Mo oo pulse 

Mod 

1.644.1 

2,749.0 

2.156.0 

1.31 

AIM-9L 

Sidewinder 

Mod 

655.1 

1,6094 

1491.4 

2.12 

AIM-9M 

Sidewinder 

Mod 

701.4 

1,321.4 

709.5 

1.01 

AIM-54A 

Phoenix 

New 

2,049.2 

2,724.6 

2,768.0 

1.35 

AIM-54C 

Phoenix 

Mod 

628.7 

3,358.7 

1 461.9 

Z01 

AIM-120A 

AMRAAM 

New 

7,650.4 

8,281.2 

LD. 

LD. 

FIM-92A 

Stinger-Basic 

New 

1,086.9 

2,971.4 

1475.3 

1.45 

FIM-92A 

Stinger-POST/RMP 

Mod 

I.U. 

I.D. 

LD. 

LD. 

AGM-65D/F/G 

OR  Maverick 

Mod 

2440.7 

6439.5 

3431.7 

1.58 

A/R/UGM 

-84A/OD 

Harpoon 

New 

1,345.3 

4,639.0 

3466.9 

1.93 

AGM-88A 

HARM 

New 

2,784.2 

4,022.9 

3,866.0 

1.39 

AGM-U4A/B 

Hellfire 

New 

510.1 

1,347.3 

819.5 

1.61 

BGM-71A 

TOW  I 

New 

2,665.5 

3,025.4 

LD. 

l.D. 

BGM-71D 

TOWQ 

Mod 

2453.1 

2,434.7 

2,449.2 

0.96 

- 

MLRS 

New 

3,782.4 

4,345,0 

I.D. 

I.D. 

M-712 

Copperhead  CLGP 

New 

1,857.8 

1,447.2 

4,150.9 

2.23 

- 

5"  Deadeye  SALGP 

New 

539.8* 

Cancelled 

Cancelled 

Cancelled 

Low 
High 
Mean 
Mean  for 
Quantity 
Adjusted  Sample 


5 10.1 

1,321.4 

709.5 

0.96 

7,650.4 

8,2814 

4,150.9 

2.23 

2498.6 

3,328.8 

2453.9 

1.58 

1446.4 

2,930.4 

2453.9 

1.58 

I.U.  -  Information  unavailable. 

I.O.  -  Insufficient  data  for  estimation  of  cost  curve. 

*  •  Excluded  from  the  mean  calculation  because  of  subsequent  cancellation. 
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Notes:  Numbers  in  parentheses  are  sample  sizes.  Bullets  (  • )  incicate  the  means. 

Figure  IV-16.  Ranges  of  Planned  and  Adjusted  Estimates 
of  Production  Costs  for  Each  Category  of  Munition 
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Figure  IV-17.  Range  of  Production  Cost  Growth  Factors 
for  Each  Category  of  Munition 

Higher  production  cost  growth  was  directly  related  to  higher  development  schedule 
growth.33  Slippage  in  the  development  schedule  involved  problems  that  also  affected 
producibility  for  several  of  the  munitions  in  the  sample,  as  discussed  in  the  case  studies  in 
Volume  II  for  the  AIM-9L  Sidewinder  and  the  Copperhead. 

However,  production  cost  growth  was  not  related  to  development  quantity  growth 
or  to  development  cost  growth,  nor  did  production  cost  growth  show  any  significant 
improvement  over  time  for  the  munitions  in  the  sample. 34 

Production  cost  growth  was  not  significantly  related  to  the  application  during  the 
development  and  production  phases  of  any  of  the  acquisition  policies.  None  of  the 
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differences  shown  in  Figure  IV- 18  are  statistically  significant 35  In  particular,  multi-year 
contracting  and  production  competition  had  no  apparent  effects  on  production  cost  growth. 
Nor  did  any  of  the  initiatives  during  the  development  phase  such  as  advanced  development 
prototypes,  independent  cost  estimates,  or  independent  listing,  significantly  reduce  the 
uncertainties  as  to  what  production  costs  would  be.  During  the  development  phase,  efforts 
were  focused  primarily  on  development  of  operational  munitions  without  much  attention  to 
either  the  producibility  of  the  munitions  or  the  development  and  testing  of  the  production 
processes,  as  discussed  in  the  case  studies  in  Volume  II  for  the  AIM-7E  Sparrow,  AIM-54 
Phoenix,  Harpoon,  HARM,  Copperhead,  and  5"  Deadeye  SALGP. 
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Figure  IV-18.  Hanga  of  Production  Cost  Growth  Factors 
for  Applications  of  Each  Acquisition  Policy 
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Figure  IV-18.  Range  of  Production  Cost  Growth  Factors 
for  Applications  of  Each  Acquisition  Policy  (Continued) 


The  extent  to  which  production  cost  growth  was  related  to  the  risks  described  in 
Chapter  HI  is  shown  in  Figure  IV- 19.36  Production  cost  growth  was  not  related  to  the 
required  levels  of  technological  advance  for  performance,  materials,  or  production 
processes.  Higher  technological  risks  did  not  necessarily  result  in  higher  production  cost 
growth.  Furthermore,  production  cost  growth  was  not  significantly  related  to  the 
percentage  requirements  for  new  test  equipment,  facilities,  or  tooling. 37 
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Figure  IV-19.  Range  of  Production  Cost  Growth  Factors 
for  Each  Category  of  Program  Risk 
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Production  cost  growth  was  directly  related  to  the  production  funding  macro¬ 
stability  measure  discussed  in  the  previous  chapter.  Higher  overall  defense  procurement 
TOA  growth  rates  were  accompanied  by  higher  production  cost  growth.  However, 
production  cost  growth  was  not  significantly  affected  by  differences  between  the  funding 
levels  in  the  congressional  appropriations  and  the  presidential  budgets  (the  procurement 
funding  micro-stability  measure  described  in  Chapter  m).38  Production  cost  growth  was 
directly  correlated  to  the  extent  of  the  overlap  between  development  and  production:  the 
greater  the  overlap,  the  greater  the  production  cost  growth.39 

F .  PRODUCTION  QUANTITY  GROWTH 

The  development  estimates  and  the  current  estimates  of  (or  actual)  production 
quantities,  and  the  production  quantity  growth  factors  are  shown  in  Table  IV-9.  The  5" 
Deadeye  SLGP  is  excluded  from  the  sample  because  the  program  was  cancelled  before  it 
entered  production.  The  production  quantity  growth  factor  for  a  munition  is  defined  as  the 
current  estimate  of  the  total  production  quantity  from  the  most  recent  S  AR  divided  by  the 
development  estimate  as  of  Milestone  II.  A  production  quantity  growth  factor  was  not 
calculated  for  the  AMRAAM  because  of  uncertainty  about  the  currently  planned  production 
quantity. 

Ranges  of  the  estimates  of  total  production  quantities  made  at  the  start  of 
development,  and  the  current  estimates  of  production  quantities,  are  shown  in  Figure  IV-20 
for  each  category  of  munition.^  The  patterns  in  Figure  IV-20  reflect  the  very  large 
development  and  current  estimates  of  the  production  quantities  of  the  Army  TOW  I,  TOW 
n,  and  MLRS  surface-launched,  surface  attack  munitions. 

The  ranges  of  production  quantity  growth  factors  for  each  category  of  munition  are 
shown  in  Figure  IV-21  4*  Production  quantity  growth  did  not  vary  significantly  between 
any  of  the  categories.  The  wide  range  in  production  quantity  growth  factors  for  most  of  the 
categories  suggests  that  not  too  much  credence  should  be  given  the  originally  planned 
estimates  of  total  production  quantities. 

Production  quantity  growth  did  not  vary  significantly  between  munitions  whose 
operational  and  technical  requirements  were  satisfied  and  munitions  for  which  those 
requirements  were  not  totally  satisfied,  as  shown  in  Figure  IV-21.  The  fact  that  production 
quantities  were  increased  for  eight  of  the  ten  munitions  that  did  not  totally  satisfy  those 
requirements  suggests  that  the  requirements  may  have  been  unduly  stringent. 


Table  IV-9.  Production  Quantities  and  Production  Quantity  Growth  Factors 
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I.U.  -  Information  unavailable. 

*  -  Excluded  from  calculation  of  mean. 
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Figure  IV-20.  Ranges  of  Planned  and  Current  Estimates 
of  Production  Quantities  for  Each  Category  of  Munition 


Figure  IV-21.  Range  of  Production  Quantity  Growth  Factors 
tor  Each  Category  of  Munition 


Production  quantity  growth  had  no  relationship  to  production  cost  growth  for  the 
munitions  in  the  sample.  A  munition  with  high  production  cost  growth  was  as  likely  to 
have  high  production  quantity  growth  as  was  a  munition  with  low  production  cost 

growth  42 

Furthermore,  the  success  of  the  development  program,  as  measured  by 
development  schedule  growth  or  development  cost  growth  had  little  or  no  measurable 
impact  on  production  quantity  growth.  These  results  are  to  be  expected,  in  that  once  a 
munition  has  been  fully  developed,  the  total  production  quantities  will  be  dictated  by 
projected  combat  requirements,  and  the  perceived  requirements  themselves  may  change 
over  time.  These  requirements  were  increased  over  time,  as  shown  by  a  significant  direct 
relationship  between  production  start  dates  and  production  quantity  growth  factors.4^ 

Production  quantity  growth  was  significantly  related  to  the  application  of  several 
acquisition  policies  during  the  production  phase,  as  shown  in  Figure  IV-22.44  Average 
production  quantity  growth  was  higher  for  munitions  with  either  system  or  subsystem 
production  competition  than  for  munitions  with  non-competitive  production.  Average 
production  quantity  growth  was  lower  for  munitions  with  foreign  production  than  for 
munitions  without  foreign  production.  Foreign  production  may  have  taken  the  place  of 
domestic  production  for  foreign  military  sales. 

The  extent  to  which  production  quantity  growth  was  related  to  the  risks  described  in 
Chapter  HI  is  shown  in  Figure  IV-23 .45  Production  quantity  growth  was  not  related  to  the 
required  levels  of  technological  advance  for  performance,  materials,  or  production 
processes,  nor  was  it  related  to  the  percentage  requirements  for  new  test  equipment, 
facilities,  or  tooling.4^  Production  quantity  growth  was  not  significantly  related  to  either  of 
the  procurement  funding  stability  measures  discussed  in  the  previous  chapter.^ 

G.  PRODUCTION  STRETCHOUT 

The  development  estimates  and  current  estimates  of  (or  actual)  production  end  dates 
and  production  spans,  and  the  production  stretchout  factors  are  shown  in  Table  IV- 10.  The 
5"  Deadeye  SALGP  is  excluded  because  the  program  was  cancelled  before  it  entered 
development 

The  current  estimates  of  production  end  dates  for  many  of  the  munitions  extend  far 
beyond  the  development  estimates.  This  happened  for  three  separate  reasons.  The  first  is 
that  production  was  started  later  than  originally  planned  because  of  slippage  in  the 
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Figure  IV-22.  Range  of  Production  Quantity  Growth  Factors 
for  Applications  of  Each  Acquisition  Policy 
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Figure  IV-23.  Range  of  Production  Quantity  Growth  Factors  for  Each  Category  of  Program  Risk 


Table  IV-10.  Production  End  Dates,  Production  Spans,  and  Production  Stretchouts 
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I.U.  -  Infonnation  unavailable. 


development  schedule,  as  discussed  in  the  second  section  of  this  chapter.  The  second 
reason  for  extended  production  end  dates  is  that  production  quantities  increased  beyond 
what  was  originally  planned,  as  discussed  in  the  previous  section  of  this  chapter.  This 
increase  would  result  in  an  increase  in  the  production  span  if  production  rates  were  not 
changed.  Production  spans  did  increase  for  all  but  one  of  the  munitions  in  the  sample,  as 
shown  in  Table  IV- 10. 

The  final  reason  for  extended  production  end  dates,  and  a  reason  for  growth  in  the 
production  spans  over  what  was  originally  planned,  is  that  production  rates  were  reduced 
over  the  originally  planned  rates,  resulting  in  production  stretchouts.  The  production 
stretchout  factor  was  calculated  as  the  production  span  growth  factor  divided  by  the 
production  quantity  growth  factor.48  a  production  stretchout  factor  was  not  calculated  for 
the  AMRAAM  because  of  uncertainty  about  the  currently  planned  production  quantity. 

The  range  of  production  stretchout  factors  for  each  category  of  munition  is  shown 
in  Figure  IV-24.49  Production  stretchout  did  not  vary  significantly  between  any  of  the 
categories.  Nor  did  production  stretchout  vary  significantly  between  munitions  whose 
operational  and  technical  requirements  were  satisfied  and  munitions  for  which  those 
requirements  were  not  totally  satisfied. 

Production  stretchout  was  inversely  related  to  production  quantity  growth:  the 
lower  the  production  quantity  growth,  the  higher  the  production  stretchout.50  Production 
stretchouts  may  have  been  used  to  retain  active  production  capacity  when  either  increases  in 
production  quantities  or  continuation  of  existing  production  rates  could  not  be  justified. 

There  was  a  positive  correlation  between  production  stretchout  and  production  cost 
growth,  but  that  correlation  is  not  statistically  significant.51 

Production  stretchout  was  inversely  correlated  with  when  the  munition  entered 
production.52  This  inverse  correlation  and  the  significant  positive  correlation  between 
production  quantity  growth  and  when  the  munition  entered  production  are  related  to  the 
very  strong  inverse  correlation  between  production  stretchout  and  production  quantity 
growth.  The  defense  buildup  in  the  first  half  of  the  1980s  resulted  in  lower  stretchouts  as 
well  as  higher  production  quantity  growth. 

Production  stretchout  was  significantly  related  to  two  acquisition  policies  applied 
during  the  production  phase,  as  shown  in  Figure  IV-25.53  Why  production  stretchout 
tended  to  be  lower  for  munitions  with  competitive  subsystem  production  is  unclear. 
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Figure  IV-24.  Range  of  Production  Stretchout  Factors  for  Each  Category  of  Munition 
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Figure  IV-25.  Range  of  Production  Stretchout  Factors 
for  Applications  of  Each  Acquistion  Policy 


Production  stretchout  tended  to  be  higher  for  munitions  with  foreign  production 
than  for  missiles  without,  and  the  difference  is  statistically  significant  As  shown  on 
Figure  IV-22  in  the  previous  section,  production  quantity  growth  tended  to  be  lower  for 
munitions  with  foreign  production.  The  greater  production  stretchouts  for  munitions  with 
foreign  production  is  a  reflection  of  the  strong  inverse  relationship  between  production 
quantity  growth  and  production  stretchout 

The  extent  to  which  production  stretchout  was  related  to  the  risks  described  in 
Chapter  in  is  shown  in  Figure  IV-26.54  Production  stretchout  was  not  related  to  the 
required  levels  of  technological  advance  for  performance,  materials,  or  production 
processes,  nor  was  it  related  to  the  percentage  requirements  for  new  test  equipment, 
facilities,  or  tooling.55  Production  stretchout  was  not  related  to  either  of  the  funding 
stability  measures  nor  to  the  development/production  overlap  ratios  discussed  in  the 
previous  chapter. 56 

H .  TOTAL  PROGRAM  COST  GROWTH 

The  estimates  of  total  program  costs  made  at  the  start  of  development,  the  current 
estimates  of  total  program  costs,  what  the  total  program  costs  would  be  if  the  development 
estimate  quantities  were  produced,  and  the  total  program  cost  growth  factors  are  shown  in 
Table  IV- 11.57  The  development  estimates  of  total  program  costs  are  from  the  earliest 
available  SARs  for  the  munitions,  subsequently  escalated  from  then-year  dollars  to  1989 
dollars.  The  current  estimates  are  from  the  latest  SARs,  also  escalated  to  1989  dollars. 
The  quantity-adjusted  total  program  costs,  based  on  the  quantity-adjusted  production  costs 
shown  in  Section  F  of  this  chapter,  allow  comparison  with  the  development  estimates  of 
total  program  costs  for  the  same  production  quantities.  The  total  program  cost  growth 
factor  is  the  quantity-adjusted  total  program  cost  divided  by  the  development  estimate  of 
total  program  cost.  Quantity-adjusted  total  program  costs  could  not  be  estimated  for  the 
A/RIM-7E  and  AIM-7F  Sparrow,  the  AMRAAM,  the  Stinger-POST/RMP,  the  TOW  I,  or 
the  MLRS  because  of  insufficient  data.  Development  estimate  program  costs  only  are 
shown  for  the  5M  Deadeye  SALGP  because  that  procurement  program  was  cancelled  before 
production  started. 

Current  estimate  total  program  costs  averaged  $3.3  billion  in  1989  dollars  for  the 
twelve  munitions  for  which  quantity-adjusted  production  costs  could  be  calculated. 
Quantity-adjusted  total  program  costs  averaged  $2.7  billion  in  1989  dollars,  about  $0.6 
billion  less  than  the  average  of  the  current  estimates.  This  difference  is  due  to  changes  in 
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Figure  IV-26.  Range  of  Production  Stretchout  Factors 
for  Each  Category  of  Program  Risk 


the  planned  or  actual  production  quantities  since  the  programs  were  initiated.  The  ranges  of 
development  estimates  and  quantity-adjusted  total  program  costs  show  large  differences 
between  categories  of  munitions,  as  shown  in  Figure  IV-27.58  The  largest  absolute  and 
percentage  differences  are  between  intercept  and  surface  attack  munitions.  The  average 
total  program  costs  for  modifications  are  approximately  two-thirds  of  those  for  the  new 
munitions  in  the  sample.  This  is  because  development  costs,  and  production  quantities  and 
costs  tended  to  be  much  less  for  the  modifications  in  the  sample. 

Table  IV-11.  Total  Program  Costs  and  Total  Program  Cost  Growth  Factors 
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Figure  IV-27.  Range  of  Planned  and  Adjusted  Estimates 
of  Total  Program  Costs  for  Each  Category  of  Munition 


For  each  of  the  categories  of  munitions,  the  average  quantity-adjusted  total  program 
cost  was  higher  than  the  average  development  estimate  total  program  cost  Total  program 
costs  increased  by  an  average  of  54  percent  for  the  twelve  munitions  for  which  quantity- 
adjusted  production  costs  could  be  calculated.  As  shown  in  Figure  IV-28,  there  is  a 
remarkable  degree  of  uniformity  for  the  total  program  cost  growth  factors  between  the 
categories  of  munitions,  with  no  statistically  significant  differences  between  the 

categories.59 
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Notes:  Numbers  in  parentheses  are  sample  sizes.  Bullets  (  * )  indicate  the  means. 

Figure  IV-28.  Range  of  Total  Program  Cost  Growth  Factors 
for  Each  Category  of  Munition 

Total  program  cost  growth  was  directly  correlated  with  production  cost  growth, 
with  a  very  high  level  of  statistical  significance.59  Total  program  cost  growth  was  not 
correlated  with  development  cost  growth,  which  is  consistent  with  the  absence  of  any 
correlation  between  production  cost  growth  and  development  cost  growth,  as  previously 
discussed.60 
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V.  FINDINGS  AND  RECOMMENDATIONS 


This  chapter  presents  the  lessons  learned  from  the  analyses  in  the  preceding 
chapters  and  from  the  case  studies  in  Volume  H  Recurring  pitfalls  leading  to  problems  are 
identified,  and  recommendations  are  made  for  avoiding  them.  To  provide  a  context  for  this 
discussion,  the  chapter  opens  with  a  summary  of  the  outcomes  of  the  acquisition  programs 
for  the  munitions  in  the  sample. 

A.  SUMMARY  OF  ACQUISITION  PROGRAM  OUTCOMES 

The  acquisition  program  schedules,  costs,  and  quantities  are  summarized  in  Table 
V-l  for  the  nineteen  munitions  in  the  sample.  Advanced  development  averaged  41  months 
in  duration  and  full  scale  development  averaged  85  months.  There  was  large  variability 
about  these  averages,  leaving  considerable  uncertainty  about  how  long  each  of  these  phases 
of  the  development  program  should  take.  The  average  full  scale  development  schedule  for 
modified  munitions  was  almost  one  year,  14  percent  longer  than  for  new  munitions.  Both 
the  development  quantity  and  development  cost  for  modified  munitions  were  approximately 
one-third  of  the  corresponding  numbers  for  new  munitions.  Development  costs  accounted 
for  approximately  6  percent  of  the  total  program  costs  for  modified  munitions  and 
approximately  15  percent  for  new  munitions.  The  average  total  program  cost  for  the 
modified  munitions  was  approximately  75  percent  of  the  average  total  program  cost  for  the 
new  munitions,  but  the  average  production  quantity  for  the  modified  munitions  was 
approximately  40  percent  of  that  for  new  munitions.  The  result  is  that  the  average  unit  cost 
for  the  modified  munitions  ($93,782)  was  almost  twice  as  high  as  for  new  munitions 
($50,271).  These  numbers  suggest  that  acquisition  programs  for  modifications  require 
comparable  management  procedures  and  the  same  degree  of  management  attention  as  are 
required  for  the  acquisition  programs  for  new  munitions. 

These  patterns  are  reinforced  by  the  acquisition  program  outcome  measures,  which 
are  summarized  in  Table  V-2.  Quantity- adjusted  total  program  cost  increased  by  an  average 
of  54  percent  over  what  was  originally  planned.  Total  program  cost  growth  differed  very 
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little  between  any  of  the  categories  of  munitions.  The  relationships  between  program 
outcomes  and  the  effects  of  the  various  risks  and  acquisition  policies  applicable  to  each 
munition  are  discemable  only  when  program  outcomes  are  disaggregated  into  more  detailed 
measures  and  data  are  obtained  from  detailed  studies  of  individual  munitions. 

Development  schedules  slipped  by  an  average  of  48  percent  over  what  was 
planned,  and  no  statistically  significant  differences  were  seen  between  categories  of 
munitions.  Development  schedule  growth  was  not  directly  related  to  any  of  the  measures 
of  risk,  or  to  any  of  the  acquisition  policy  initiatives.  Development  schedule  growth  was 
affected  by  problems  that  subsequently  affected  production  costs. 

Development  quantity  growth  and  development  cost  growth  were  directly  related 
for  the  munitions  in  the  sample,  but  neither  measure  was  related  to  development  schedule 
growth.  Development  cost  growth  was  significantly  greater  for  intercept  missiles  than  for 
surface  attack  missiles,  while  development  quantity  growth  was  significantly  greater  for 
air-launched  than  for  surface-launched  munitions,  and  for  modified  munitions  than  for  new 
munitions.  The  case  studies  in  Volume  II  indicate  that  the  original  estimates  of 
development  quantities  and  costs  (and  difficulties)  were  overly  optimistic.  Both 
development  quantity  growth  and  development  cost  growth  were  significantly  lower  for 
munitions  that  had  advanced  development  phases  and  advanced  development  prototypes. 
Development  quantity  growth  was  also  significantly  lower  for  munitions  that  had 
independent  cost  analysis.  Development  cost  growth  was  significantly  lower  for  munitions 
that  had  competition  at  the  subsystem  level  during  full  scale  development  Development 
cost  growth  was  inversely  related  to  the  percentage  requirements  for  new  facilities; 
munitions  with  higher  development  cost  growth  tended  to  have  lower  percentages  of  floor 
area  in  new  facilities. 

Production  costs  for  the  development  estimate  quantities  increased  by  an  average  of 
58  percent  over  what  was  originally  planned.  Production  cost  growth  was  not  significantly 
different  between  any  of  the  categories  of  munitions.  Production  cost  growth  was  not 
related  to  either  development  cost  growth  or  development  quantity  growth,  but  was  directly 
related  to  development  schedule  growth.  Producibility  problems  that  resulted  in  increased 
production  costs  appear  to  have  surfaced  during  development,  causing  slippage  in  the 
development  schedule.  Production  cost  growth  was  directly  correlated  to  both 
procurement  budget  growth  ("procurement  funding  macro-stability"),  and  the  extent  of 
overlap  between  full  scale  development  and  production.  Production  cost  growth  did  not 
differ  significantly  with  any  of  the  other  measures  of  program  risk,  or  with  the  applicability 


of  any  of  the  acquisition  program  initiatives.  Specifically,  production  cost  growth  was  not 
affected  by  multi-year  contracting,  or  by  competition  at  either  the  system  or  subsystem 
levels  of  production. 

Production  quantities  increased  by  an  average  of  42  percent  over  what  were 
originally  planned.  Again,  there  was  large  variability  about  the  average.  Little  or  no 
credence  should  be  given  early  estimates  of  planned  production  quantities.  Production 
quantity  growth  did  not  differ  significantly  between  any  of  the  munition  categories.  Even 
though  average  production  quantity  growth  for  new  and  modified  munitions  differed  by 
approximately  a  factor  of  two,  the  difference  is  not  statistically  significant  because  of  the 
extreme  variability  about  the  two  averages.  Production  quantity  growth  was  not  related  to 
production  cost  growth.  Production  quantity  growth  was  significantly  greater  for 
munitions  with  system  or  subsystem  competition,  and  for  munitions  without  foreign 
production.  As  noted  in  several  of  the  case  studies  in  Volume  II,  production  competition 
becomes  more  feasible  with  larger  production  quantities.  Foreign  production  reduced  the 
domestic  production  below  what  it  might  otherwise  have  been. 

Production  of  the  originally  planned  quantities  stretched  out  an  average  of  119 
percent  over  the  original  schedule.  Production  stretchout  was  not  significantly  different 
between  any  of  the  munition  categories.  Production  stretchout  had  a  significant  inverse 
relationship  with  production  quantity  growth,  but  a  statistically  significant  direct 
relationship  with  production  cost  growth  could  not  be  established.  As  might  be  expected 
because  of  the  strongly  significant  inverse  relationship  with  production  quantity  growth, 
production  stretchout  was  less  for  munitions  with  subsystem  production  competition  and 
for  munitions  without  foreign  production.  Without  foreign  production,  there  is  an 
incentive  to  stretch  production  to  maintain  a  production  base  for  domestic  and  foreign  sales. 

All  operational  and  technical  requirements  listed  in  the  SARs  were  satisfied  for  41 
percent  of  the  seventeen  munitions  that  have  competed  development  testing.  That  59 
percent  were  deployed  without  fully  satisfying  their  operational  and  technical  requirements 
suggests  that  the  requirements  might  have  been  unduly  stringent.  This  is  further  supported 
by  no  statistically  significant  differences  in  production  quantity  growth  between  the 
munitions  that  satisfied  those  requirements  and  the  munitions  that  did  not.  None  of  the 
other  program  outcome  measures  differed  significantly  between  whether  or  not  the 
operational  and  technical  requirements  were  satisfied.  The  percentage  of  munitions 
satisfying  the  operational  and  technical  requirements  did  not  vary  significantly  among  any 
of  the  acquisition  policies  applied.  The  percentage  was  lower  for  munitions  with  extensive 


requirements  for  advances  in  performance  and  production  technologies.  Percentage 
requirements  for  new  test  equipment  were  significantly  lower  for  munitions  that  satisfied  all 
operational  and  technical  requirements. 

The  two  munitions  that  faced  substantial  changes  in  the  threat  during  development 
(HR  Maverick  and  HARM)  differed  from  the  remainder  of  the  munitions  in  the  sample  in 
that  both  had  advanced  development  phases  with  either  system  or  subsystem  prototypes, 
independent  cost  estimates,  independent  testing,  and  system  warranties.  None  of  the 
outcome  measures  for  these  two  munitions  varied  significantly  from  the  outcome  measures 
for  the  remainder  of  the  munitions  in  the  sample.  If  this  risk  had  any  effect  on  development 
quantity  growth  or  development  cost  growth,  it  may  have  been  offset  by  the  effects  of  the 
acquisition  policies  that  were  applied.  For  all  of  the  munitions  in  the  sample,  development 
quantity  growth  and  development  cost  growth  were  lower  for  munitions  which  had 
advanced  development,  prototypes,  and  independent  cost  estimates.  However,  there  is  no 
way  to  test  whether  or  not  any  effects  of  these  policies  on  development  quantity  growth  or 
development  cost  growth  may  have  been  offset  by  threat  changes,  because  there  were  no 
munitions  with  threat  changes  but  without  application  of  these  policies. 

Of  the  outcome  measures  shown  in  Table  V-2,  only  two  have  shown  any 
statistically  significant  improvement  over  time.  Development  quantity  growth  and 
production  stretchout  have  both  decreased.  None  of  the  measures  has  gotten  worse. 
Production  quantity  growth  has  been  higher  for  the  munitions  that  entered  production  later. 
To  some  extent  that  is  an  artificiality,  because  for  a  number  of  the  older  munitions  in  the 
sample,  production  quantities  increased  and  then  were  reduced  when  future  planned 
production  quantities  were  transferred  to  follow-on  modifications. 

Several  other  factors  have  affected  the  outcome  measures.  The  first  has  to  do  with 
the  sample  itself.  The  sample  includes  several  new,  Army-managed,  surface  attack, 
surface-  launched  munitions  with  very  large  production  quantities.  Planned  development 
schedules,  development  quantities,  and  development  cost  estimates  for  these  munitions 
tended  to  be  conservative,  showing  less  than  average  subsequent  growth.  In  contrast, 
several  Navy-managed  modifications  of  air-launched  intercept  missiles  in  the  sample  had 
planned  development  schedules,  development  quantities,  and  development  cost  estimates 
that  tended  to  be  overly  optimistic. 

The  second  factor  affecting  the  outcome  measures  was  the  defense  buildup  of  the 
1980s.  Production  quantity  growth  was  higher  and  production  concurrency  and 


production  stretch  were  lower  for  the  munitions  that  entered  production  during  the  first  half 
of  the  1980s  when  defense  procurement  budgets  increased.  Since  production  cost  growth 
did  not  show  any  trend  with  time,  the  effect  of  the  lower  concurrency  ratios  during  that 
period  was  to  limit  production  cost  growth  to  less  than  it  would  have  been  otherwise. 

A  third  factor  affecting  the  outcome  measures  was  the  interrelated  effects  of 
technological  risk,  requirements  for  new  resources,  and  the  application  of  several  of  the 
acquisition  policies.  Munitions  with  higher  technology  and  new  resource  requirements  also 
tended  to  have  advanced  development  phases  and  independent  cost  estimates.  Munitions 
which  had  higher  technology  and  new  resource  requirements  and  which  also  had  advanced 
development  phases  and  independent  cost  estimates  had  significantly  lower  development 
quantity  growth  factors  and  development  cost  growth  factors  than  did  munitions  which  had 
low  technology  and  resource  requirements  and  which  did  not  have  advanced  development 
phases  and  independent  cost  estimates.  For  the  entire  sample,  development  quantity 
growth  was  significantly  lower  for  munitions  with  advanced  development  phases  and 
independent  cost  estimates;  development  cost  growth  was  significantly  lower  for  munitions 
with  advanced  development  phases.  Both  development  quantity  growth  and  development 
cost  growth  were  higher  for  the  one  munition  which  had  high  technology  requirements,  but 
did  not  have  an  advanced  development  phase  or  an  independent  cost  estimate,  than  for  any 
of  the  other  munitions  with  high  technology  requirements  which  did  have  advanced 
development  phases  and  independent  cost  estimates.  This  would  suggest  that  the  effects  of 
the  technology  risks  may  have  been  offset  by  advanced  development  and  independent  cost 
estimates,  but  the  sample  size  is  not  large  enough  to  test  for  statistically  significant 
differences. 

There  were  no  munitions  which  had  high  resource  requirements,  but  which  did  not 
have  advanced  development  phases  or  independent  cost  estimates.  Nor  were  there  any 
munitions  in  the  technology  and  resource  requirements  subsample  which  had  advanced 
development  phases  and  independent  cost  estimates,  but  which  also  had  low  requirements 
for  new  resources.  As  a  result,  the  effects  on  development  quantity  growth  and 
development  cost  growth  of  the  resource  requirements  risks  could  not  be  separated  from 
the  effects  of  advanced  development  and  independent  cost  estimates. 

For  a  similar  reason,  it  is  not  possible  to  separate  out  the  effects  on  production  cost 
growth  of  high  technology  and  new  resource  requirements  from  the  effects  of  low  rate 
initial  production.  There  were  not  enough  munitions  in  the  technology  and  resource 
requirements  subsample  with  either  low  requirements  for  new  technology  or  resources,  or 


which  did  not  have  low  rate  initial  production  phases,  to  test  for  statistically  significant 
differences. 

By  similar  reasoning,  the  outcomes  were  affected  by  the  interactions  between 
whether  the  munitions  were  new  or  modified  and  whether  or  not  they  underwent  advanced 
development  All  of  the  new  munitions  underwent  advanced  development  while  only  two 
of  the  modified  munitions  did.  Both  development  quantity  growth  and  development  cost 
growth  were  significantly  lower  for  munitions  that  underwent  advanced  development. 
There  was  no  way  to  test  the  effects  of  advanced  development  on  the  program  outcome 
measures  for  new  munitions.  For  the  modified  munitions  in  the  sample,  development 
quantity  growth  and  development  cost  growth  averages  were  lower  by  half  for  the  modified 
munitions  that  underwent  advanced  development  than  for  those  that  did  not,  but  because  of 
the  small  sample  size,  it  was  not  possible  to  show  that  these  differences  were  statistically 
significant 

The  effect  of  competition  on  the  program  outcome  measures  was  limited. 
Competition  in  the  advanced  development  phase  had  no  discernible  effect  on  development 
schedule,  quantity,  or  cost  growth.  None  of  the  munitions  in  the  sample  had  competition  at 
the  system  level  during  full  scale  development  Development  cost  growth  was  significantly 
lower  for  munitions  with  competition  at  the  subsystem  level  during  full  scale  development, 
but  neither  development  schedule  nor  development  quantity  growth  were  affected. 
Production  quantity  growth  was  greater  for  munitions  with  production  competition,  but  as 
noted  above,  the  direction  of  causation  is  not  clear. 

Finally,  two  other  acquisition  policies  --  independent  testing  and  multi-year 
contracting  --  had  no  discemable  effects  on  any  of  the  outcome  measures.  The  lack  of 
statistically  significant  differences  in  (1)  any  program  outcomes  for  these  two  policies,  and 
(2)  in  some  program  outcomes  for  other  policies,  may  be  due  to  the  small  sample  size.  A 
larger,  better  stratified  sample  might  uncover  more  statistically  significant  relationships. 

B  .  PITFALLS:  MAJOR  PROBLEMS  AND  WHY  THEY  RECUR 

Several  recurring  problems  were  observed  from  the  analyses  in  the  preceding 
chapters  and  from  the  case  study  examples  of  Volume  II  that  affected  the  outcomes  of  the 
acquisition  programs  for  the  munitions  in  the  sample. 


1 .  Recurring  Pitfall  No.  1 

The  time  and  funds  necessary  for  development  of  modifications  were  consistently 
underestimated.  Development  testing  requirements  were  optimistic  in  terms  of  both  the 
numbers  of  test  articles  required  and  the  test  schedule.  For  the  AIM-7F  Sparrow 
modification,  4  months  were  planned  for  technical  evaluation,  but  22  months  were  actually 
required;  3  months  were  planned  for  operational  evaluation,  but  30  months  were  actually 
required.  Technological  difficulties  were  seriously  underestimated  for  the  AIM-9L 
Sidewinder  modification  —  33  months  for  development,  30  test  articles,  and  a  high  degree 
of  concurrency  between  development  and  production  were  originally  planned. 
Subsequently,  the  development  program  was  drastically  revised.  The  technological 
requirements  for  the  change  from  analog  to  digital  components  in  the  guidance  and  control 
section  modification  of  the  AIM-54C  Phoenix  were  also  underestimated.  The  amount  of 
time  required  to  develop  the  imaging  infra-red  seeker  for  the  HR  Maverick  was  grossly 
underestimated,  as  were  the  development  costs  for  the  TOW  n. 

The  result  was  development  schedule  growth  and  development  cost  growth  for 
modifications  being  as  high  as  for  new  weapons,  and  development  quantity  growth  for 
modifications  being  much  higher  than  for  new  weapons.  This  is  primarily  because  most  of 
the  modifications  involved  large  changes  to  the  sensors,  software,  avionics,  and  other 
electronics  subsystems,  which  comprise  over  60  percent  of  the  cost  of  most  munitions. 

2.  Recurring  Pitfall  No.  2 

Technological  uncertainties  were  not  adequately  identified  early  in  the  development 
programs.  Technology  availability  was  consistently  overestimated.  This  was  reflected  in 
large  growth  in  development  schedules  and  quantities  and  in  development  and  production 
costs.  Technological  difficulties  associated  with  the  development  of  the  pulse-Doppler 
seeker  and  guidance  system  for  the  AIM-7F  Sparrow,  and  the  all-aspect  seeker  for  the 
AIM-9L  Sidewinder  did  not  surface  until  well  into  the  development  program.  The 
technological  requirements  for  a  long-range  launch-and-forget  missile  for  fleet  air  defense 
were  originally  underestimated  in  the  AIM-54A  Phoenix  program,  as  were  the  subsequent 
requirements  for  the  solid-state  modification  of  the  AIM-54C.  The  technological 
requirements  of  the  AMRAAM  were  originally  assessed  in  comparison  to  the  AIM-7 
Sparrow,  but  were  much  more  similar  to  what  the  requirements  would  be  for  a  miniaturized 
Phoenix.  The  Air  Force’s  all-weather,  day  and  night  operational  requirement  for  the  IIR 
Maverick  exceeded  what  could  be  done  with  the  technology  available  at  the  start  of  full 


scale  development.  The  technological  requirements  of  the  5"  Deadeye  SALGP  were 
originally  assessed  in  comparison  to  the  Copperhead  without  adequate  consideration  of  the 
extensive  miniaturization  (to  17  percent  of  the  volume  in  Copperhead)  that  would  be 
required  for  the  guidance  and  control  electronics.  In  contrast,  the  technological 
requirements  were  well-  identified,  and  the  technology  was  available  at  the  start  of 
development  for  both  the  Hellfire's  semi-active  laser  seeker  and  for  the  MLRS. 

A  risk  management  system  is  needed  during  program  execution  to  track  and  obtain 
feedback  on  technical  trends,  as  well  as  schedule  and  cost  trends.  Important  technological 
ingredients  in  the  program  need  to  be  quantified  with  respect  to  the  trends  in  technology 
over  time  in  order  to  understand  how  far  the  state-of-the-art  is  being  pushed.  Technical 
trends  over  time  are  not  well-understood  outside  of  missile  propulsion  systems  and  aircraft 
engines  and  airframes.  Similar  analyses  need  to  be  done  for  software,  guidance  and 
control  subsystems,  warheads,  and  production  processes.  The  availability  of  such 
information  would  be  of  great  value  in  determining  the  requirements  for  advanced 
development  prototypes  and  exploratory  development  testing. 

3.  Recurring  Pitfall  No.  3 

Design  and  production  concepts  were  inadequately  demonstrated  and  evaluated  in 
several  of  the  development  programs.  Advanced  development  prototyping  was  not  always 
successful  in  eliminating  the  major  technological  uncertainties  for  the  munitions  in  the 
sample.  Prototyping  was  generally  restricted  to  the  munition  itself,  but  even  in  this 
restricted  sense,  it  was  not  always  totally  successful  in  eliminating  technological 
uncertainties.  For  example,  the  technology  embodied  in  advanced  development  prototypes 
for  the  AMRAAM  differed  substantially  from  what  finally  evolved  during  full  scale 
development:  the  solid-state  transmitter  in  the  prototype  was  not  capable  of  satisfying  the 
operational  and  technical  requirements  of  the  missile  and  had  to  be  replaced  with  a 
transmitter  based  on  traveling-wave-tube  technology,  which  required  extensive 
development  efforts  for  miniaturization  and  cooling.  For  that  missile,  the  competitive 
advanced  development  prototypes  served  more  as  a  basis  for  contractor  selection  than  as  a 
means  for  reducing  technological  uncertainty.  The  original  plan  for  testing  10  missiles 
from  each  contractor  was  reduced  to  5  missiles  from  Raytheon  and  3  from  Hughes.  The 
testing  of  the  advanced  development  prototypes  did  not  adequately  demonstrate  that  the 
technology  was  in  hand.  In  contrast,  the  competitive  advanced  development  of  the  MLRS 
produced  prototypes  that  achieved  good  results  during  testing,  thereby  demonstrating  that 


the  technology  was  in  hand.  The  competitive  advanced  development  of  the  Hellfire  had 
similar  beneficial  effects  on  limiting  development  cost  growth. 

Prototyping  of  production  processes  was  not  generally  done  for  the  munitions  in 
the  sample,  either  in  advanced  development  or  in  full  scale  development  The  result  was 
lower-than-expected  yield  levels  from  new  production  processes,  resulting  in  high 
production  cost  growth  for  the  HARM  and  the  Copperhead.  Uncertainties  as  to  production 
process  feasibility  and  cost  as  well  as  uncertainties  as  to  the  munition's  technical  and 
performance  feasibility  and  cost,  need  to  be  addressed  and  reduced  during  advanced 
development 

4.  Recurring  Pitfall  No.  4 

Producibility  was  not  adequately  considered  in  the  design  decisions  for  several  of 
the  munitions  in  the  sample.  This  resulted  in  producibility  and  quality-control  problems, 
which  caused  production  costs  to  increase.  Producibility  and  quality-control  problems  with 
the  AIM-54C  resulted  in  high  production  cost  growth  and  the  Navy's  non-acceptance  of 
deliveries  in  June  1984.  Fabrication  and  quality-control  problems  resulted  in  half  of  the 
first  20  AMRAAM  development  test  missiles  being  returned  to  the  contractor  because  of 
defects.  Completion  of  a  compressed  development  schedule  for  the  Harpoon,  in  order  to 
counter  an  immediate  threat,  was  emphasized  to  the  detriment  of  producibility,  resulting  in 
higher  than  average  production  cost  growth.  Producibility  problems  with  the  circuit  boards 
and  seeker  for  the  HARM  required  much  more  extensive  testing  with  expensive  labor  than 
was  anticipated.  Production  planning  for  the  Copperhead  was  initiated  approximately  two 
years  after  the  start  of  full  scale  development,  too  late  to  make  any  substantial  design 
changes  to  enhance  producibility,  resulting  in  the  highest  production  cost  growth  of  any  of 
the  munitions  in  the  sample.  More  emphasis  needs  to  be  placed  on  understanding  early  in 
development  the  cost  of  producing  the  article  in  quantity. 

5.  Recurring  Pitfall  No.  5 

Unrealistic  test  planning  contributed  to  development  schedule  growth,  development 
cost  growth,  and  product  unreliability  for  several  of  the  munitions  in  the  sample.  For 
modified  munitions,  estimates  of  the  numbers  of  test  articles  that  would  be  required  were 
woefully  inadequate.  For  the  intercept  and  anti-radar  munitions,  estimates  of  the  time  that 
would  be  required  for  each  firing  were  generally  overly  optimistic  because  of  inadequate 
consideration  or  knowledge  of  competing  requirements  by  other  development  programs  for 


limited  test  resources.  The  AEM-54A  Phoenix  development  schedule  was  delayed  by 
limited  availability  of  F-111B  test  aircraft  and  delays  in  the  F-14A  flight  test  program. 
Testing  of  the  AMRAAM  has  been  extensively  delayed  by  problems  with  weather,  targets, 
and  launch  aircraft,  and  by  funding  limitations  for  a  third  test  site.  An  optimistic  test 
schedule  for  the  HARM  was  subsequently  constrained  to  fewer  than  die  planned  number  of 
firings  because  of  limitations  in  the  availability  of  test  facilities,  particularly  for  testing  of 
software. 

Slippage  in  the  test  schedule  for  the  AIM-54A  Phoenix  led  to  a  reduction  in  the 
development  quantity  in  order  to  save  on  development  costs  and  limit  the  slippage  in  the 
development  schedule.  Development  quantities  were  also  reduced  below  originally  planned 
levels  in  order  to  save  on  the  development  costs  for  the  HR  Maverick  and  the  Copperhead. 
The  result  was  that  reliability  problems  were  not  discovered  until  initial  production. 

6.  Recurring  Pitfall  No.  6 

Procurement  plans  with  larger  overlaps  between  development  and  production  were 
generally  overtaken  by  unforeseen  problems,  and  had  to  be  restructured  for  several  of  the 
munitions  in  the  sample.  Inadequate  recognition  was  given  to  the  possibility  of 
technological  problems  during  testing  and  early  operational  use.  As  shown  in  Chapter  IV, 
the  extent  of  development/production  overlap  was  directly  correlated  with  production  cost 
growth  for  the  munitions  in  the  sample.  Extensive  overlaps  between  development  and 
production  in  the  acquisition  programs  for  the  AIM-9L  Sidewinder,  AIM-54C  Phoenix, 
Harpoon,  and  Copperhead  were  accompanied  by  big  growths  in  production  costs. 
Because  of  early  failures  in  testing,  the  AIM-9L  Sidewinder  acquisition  program 
subsequently  had  to  be  restructured  to  allow  technical  problems  to  be  identified  and 
corrected  before  production.  Reliability  problems  with  the  AIM-54C  Phoenix  and  the 
Copperhead  subsequently  resulted  in  delayed  deliveries  of  production  munitions.  The 
simultaneous  time  frame  for  development  test  and  evaluation  and  initial  operational  test  and 
evaluation  has  contributed  to  the  high  development  cost  growth  and  extensive  development 
schedule  slippage  in  the  AMRAAM  acquisition  program.  Low  reliability  during  testing 
resulted  in  stretchout  of  TOW  I  production. 

7.  Recurring  Pitfall  No.  7 

Available  methods  for  estimating  schedules  and  costs  were  inadequate  for  most  of 
the  munitions  in  the  sample.  Estimates  of  the  time  and  resources  required  to  meet 
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technological  objectives  during  development  programs  were  not  consistent  with  trends  over 
time  in  the  technological  state-of-the-art.  Actual  times  required  for  development  of 
analogous  subsystems  were  neither  consistently  considered  nor  incorporated  into  the 
schedule  estimates.  The  tradeoffs  between  development  task  accomplishment  times  and  the 
amounts  of  development  resources  applied  were  not  well-understood  for  the  different 
categories  of  munitions  subsystems. 

Production  costs  for  the  early  part  of  the  production  phase  often  showed  Unit  1 
costs  increasing  by  a  factor  of  3- 10  with  a  steeper  learning  curve  for  many  of  the  munitions 
in  the  sample;  Unit  1  and  slope  were  consistently  misestimated.  Original  production 
schedule  and  cost  estimates  were  very  optimistic,  and  did  not  always  incorporate  the 
production  experience  of  analogous  subsystems.  Where  there  was  no  comparable 
experience  in  production,  there  was  a  potential  for  large  cost  growth  production  if  schedule 
and  cost  uncertainties  had  not  been  adequately  addressed  during  development.  More 
emphasis  needs  to  be  placed  on  understanding  early  in  development  the  cost  of  producing 
the  article  in  quantity. 

8.  Recurring  Pitfall  No.  8 

Independent  review  of  risks,  schedules,  costs,  and  test  requirements  and  results 
was  not  consistently  applied.  The  military  services  and  the  contractors  were  almost 
perpetually  optimistic  about  what  could  be  accomplished.  This  predilection  to  program 
new  systems  for  success  will  naturally  bias  the  early  cost  and  schedule  estimates.  Checks 
and  balances  are  necessary  between  OSD  and  the  military  services.  It  does  not  hurt  to  have 
a  "devil's  advocate"  on  the  staff  to  question  the  need  for  systems  and  the  critical 
assumptions  underlying  the  schedule  and  cost  estimates. 

C.  RECOMMENDATIONS 

1 .  Major  munition  modification  programs  require  the  same  types  of  management 
procedures  and  measures  to  identify  and  manage  acquisition  program  risks  as 
do  new  munition  acquisition  programs.  Modification  programs  should  be 
subjected  to  concept  exploration  and  advanced  development  phases  with 
Milestone  0  (Mission  Needs  Determination)  and  Milestone  I  (Concept 
Selection)  reviews.  Advanced  development  prototypes  of  appropriate 
subsystems  and  production  processes  should  be  required  to  adequately 
demonstrate  achievement  of  performance  and  producibility  requirements  prior 
to  entering  into  full  scale  development. 


2.  Technological  risks,  in  terms  of  performance  and  production  processes,  need 
to  be  identified  at  the  start  of  development  Technology  trending  models 
should  be  used  for  identifying  potential  risks  in  the  development  of  software, 
guidance  and  control  subsystems,  warheads,  and  production  processes.  Such 
models  have  been  successfully  developed  and  used  for  identifying 
technological  risks  of  developing  aircraft  airframe  and  propulsion  subsystems 
and  missile  propulsion  subsystems.  Similar  models  need  to  be  developed  for 
software,  guidance  and  control  subsystems,  warheads,  and  many  types  of 
production  processes. 

3.  Technological  risks,  in  terms  of  munition  performance  and  production 
processes,  need  to  be  systematically  reduced  as  early  in  development  as 
possible.  Advanced  development  prototyping  of  high-risk  performance  and 
production  technologies  should  be  required,  and  the  results  should  be 
independently  reviewed  before  the  Milestone  II  program  go-ahead  decision. 
Competitive  development  should  be  evaluated  as  a  means  to  obtain  alternative 
solutions  to  technological  problems. 

4.  Prod'  cibility  should  be  demonstrated  and  independently  reviewed  before  the 
Milestone  IIIA  production  release  decision.  Initial  production  should  be  at  a 
low  rate  until  problems  have  been  identified  and  corrected  during  early 
production  and  operational  experience.  Overlaps  between  development  and 
production  should  be  minimized  to  the  extent  possible. 

5.  Improve  schedule  and  cost-estimating  procedures  and  models  by  requiring 
systematic  identification  and  incorporation  of  uncertainties  and  time  and  cost 
tradeoffs  and  use  of  appropriate  data  bases  and  analogies.  Separate  estimates 
should  be  made  for  important  subsystems.  Where  resources  such  as  test 
ranges  and  test  vehicles  (and  crews)  are  shared  between  acquisition  programs, 
require  that  the  resource  constraints  and  other  demands  on  those  resources  be 
adequately  reflected  in  schedule  and  cost  estimates. 

6.  Emphasize  the  assessment  of  risks,  schedules,  costs,  and  test  results  at  the 
OSD  level,  independent  of  the  military  service.  Require  consistent,  full 
reporting  in  the  SARs  or  Defense  Acquisition  Executive  Summaries  (DAES)  to 
identify  the  program  risks,  describe  the  acquisition  strategy,  show  important 
subsystems  separately,  provide  a  summary  of  test  results,  provide  complete 
non-recurring/recurring  cost  splits  and  scope  changes,  and  include  joint  service 
and  foreign  military  sales  costs  and  quantities. 

Checklists  of  critical  questions  should  be  used  at  Defense  Acquisition  Board 
reviews.  A  draft  of  such  a  checklist  is  included  as  Appendix  B. 


7.  Conduct  an  on-going  systematic  effort  at  the  OSD  staff  level  to  measure 
acquisition  program  effectiveness.  Alternative  acquisition  policies  should  be 
compared,  in  a  manner  similar  to  what  has  been  done  in  this  report,  except 
across  a  wider  sample  of  acquisition  programs,  to  determine  which  policies 
have  been  effective.  Acquisition  program  histories  should  be  documented  in 
case  studies  similar  to  those  in  Volume  II  of  this  report,  to  provide  a  corporate 
memory  of  lessons  learned. 


NOTES 


II- 1 .  The  statistical  significance  of  the  relationships  between  whether  or  not  there  was 

•  an  advanced  development  phase,  competition  at  the  subsystem  level  during  full 

scale  development,  independent  testing,  or  independent  cost  estimates  was 
calculated  from  Fisher's  exact  probability  test  for  a  2x2  contingency  table  with 
small  sample  sizes: 


FSD  Subsystem  .CQmpeatiQn-fQi  Probability 

4  of  12  with  AD  and  0  of  7  without  AD  12.8% 

Independent  Testing  for  Probability 

3  of  12  with  AD  and  0  of  7  without  AD  22.7% 

Independent  CostEstimatss  to  Probability 

6  of  12  with  AD  and  0  of  7  without  AD  3.4% 


II-2.  A  test  using  the  Table  of  the  Total  Number-of-Runs  Test  for  Randomness  in  the 
Chemical  Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for 
small,  unequal  sample  sizes  was  performed  to  test  whether  or  not  subsystem 
production  competition  is  randomly  distributed  over  time.  There  are  four  runs  in 
the  data.  The  probability  of  four  or  fewer  runs  with  samples  of  ten  competitive 
•  munitions  and  seven  non-competitive  munitions  is  equal  to  .024. 


II-3.  The  statistical  significance  of  the  relationships  between  whether  or  not  there  was 
system  production  competition  and/or  subsystem  production  competition  was 
calculated  from  Fisher's  exact  probability  test  for  a  2x2  contingency  table  with 
small  sample  sizes: 

Subsystem  Production  CompetmcD-for  Probability 

9  or  10  of  12  with  system  competition  and 

0  or  1  of  6  without  system  competition  1.8% 


II-4. 


The  statistical  significance  of  the  relationships  between  whether  or  not  the 
munition  was  new  or  modified  and  whether  or  not  there  was  subsystem 
production  competition  or  low-rate  initial  production  phases  was  calculated  from 
Fisher's  exact  probability  test  for  a  2x2  contingency  table  with  small  sample  sizes: 


Subsystem  Production  Competition  for  Probability 

1  or  2  of  8  new  and  9  or  8  of  9  modified  1.3% 

Low-Rate  Initial  Production  Phase  for  Probability 

7  or  more  of  9  new  and  2  or  fewer  of  10  modified  1.9% 


Note-1 


n-5.  The  statistical  significance  of  the  relationships  between  military  service,  missile 
type,  and  whether  or  not  there  was  multi-year  contracting  was  calculated  from 
Fisher's  exact  probability  test  for  a  2x2  contingency  table  with  small  sample  sizes:  • 

Multi-Year  Contracting  for  Probability 

5 of  7  Army  and  Oof  11  other  services  02% 

0  of  12  AGM  +  AIM  and  5  of  6  other  munitions  0.1% 


m-i. 


The  statistical  significance  of  the  relationships  between  whether  or  not  there  was  a 
substantial  change  to  the  threat  and  whether  or  not  there  was  an  advanced 
development  (AD)  phase,  prototypes  during  advanced  development,  independent 
testing,  independent  cost  estimates,  and  system  warranties  was  calculated  from 
Fisher's  exact  probability  test  for  a  2x2  contingency  table  with  small  sample  sizes: 


Advanced  Development  for 

2  of  2  with  changed  threats  and  10  of  17  other  munitions 

Prototypes  During  Advanced  Development  for 
2  of  2  with  changed  threats  and  8  of  9  other  munitions 
with  AD  phases 

Independent  Testing  for 

2  of  2  with  changed  threats  and  1  of  15  other  munitions 

Independent  Cost  Estimates  for 
2  of  2  with  changed  threats  and  4  of  14  other  munitions 

System  Warranties  for 

2  of  2  with  changed  threats  and  4  of  14  other  munitions 


Probability 

38.6% 

Probability 

81.8% 

Probability 

2.2% 

Probability 

12.5% 

Probability 

12.5% 


IE-2. 


The  statistical  significance  of  the  relationships  between  the  required  level  of 
technological  advance  and  each  category  of  munition  was  calculated  from  Fisher's 
exact  probability  test  for  a  2x2  contingency  table  with  small  sample  sizes: 


Extensive  Performance  Advances,  for 

4  or  5  intercept  and  2  or  fewer  of  4  surface  attack 

3  or  fewer  of  6  air  launched  and  3  or  more  of  5  surface  launched 

5  or  6  of  7  new  and  1  or  0  of  4  modified 


Probability 

17.5%  • 

60.8% 

19.7% 


Extensive  Material  Advances  for 
0  or  1  of  5  intercept  and  4  or  3  of  6  surface  attack 
2  or  fewer  of  6  air  launched  and  2  or  more  of  5  surface  launched 
2  or  fewer  of  7  new  and  2  or  mere  of  4  modified 


Probability 

34.8% 

80.3% 

91.2% 


Extensive  Production  Advances  for  Probability 

0  or  1  of  5  intercept  and  3  or  2  of  6  surface  attack  57.6% 

0  or  1  of  6  air  launched  and  3  or  2  of  5  surface  launched  42.4% 

2  or  3  of  7  new  and  1  or  Oof  4  modified  72.1% 

Substantial  or  Extensive  Performance  Advances  for  Probability 

5  of  5  intercept  and  5  of  6  surface  attack  54.5% 

5  of  6  air  launched  and  5  of  5  surface  launched  54.5% 

6  of  7  new  and  4  of  4  modified  63.6% 


Substantial  or  Extensive  Materials  Advances  for 

5  of  5  intercept  and  5  of  6  surface  attack 

4  of  6  air  launched  and  5  of  5  surface  launched 

6  or  7  of  7  new  and  3  or  2  of  4  modified 


Probability 

54.5% 

27.3% 

89.1% 


Note-2 


Substantial  or  Extensive  Production  Advances  for  Probability 

3, 4  or  5  of  5  intercept  and  6, 5,  or  4  of  6  surface  attack  81.8% 

4, 3,  or  6  of  6  air  launched  and  5, 4,  or  3  of  5  surface  launched  81.8% 

7  of  7  new  and  2  of  4  modified  10.9% 


ra-3.  The  numbers  of  new  and  modified  munitions  for  each  level  of  required 
performance  and  materials  technology  advance  are  shown  in  the  first  table  below. 
The  numbers  of  new  and  modified  munitions  for  each  level  of  required 
performance  and  production  technology  advance  are  shown  in  the  second  table 
below.  The  munitions  are  separated  into  new  and  modified  in  these  tables 
because  of  the  differences  in  performance  technology  requirements  between  new 
and  modified  munitions. 

Numbers  of  Munitions  in  the  Sample  for  Each  Level 
of  Required  Performance  and  Materials  Technology 
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Numbers  of  Munitions  in  the  Sample  for  Each  Level 
of  Required  Performance  and  Production  Technology 
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m-4.  The  statistical  significance  of  the  relationships  between  the  required  levels  of 
technological  advance  and  whether  or  not  each  of  the  acquisition  policies  were 
applied  was  calculated  from  Fisher's  exact  probability  test  for  a  2x2  contingency 
table  with  small  sample  sizes: 


Extensive  Performance  Advances  for  Probability 

5  or  6  of  8  with  AD  and  1  or  0  of  3  without  AD  42.4% 

4  or  5  of  5  with  AD  and  1  or  0  of  3  with  non-competitive  AD  28.6% 

4  of  6  with  prototypes  in  AD  and  0  of  1  without  prototypes  in  AD  242.9% 

0  of  2  with  FSD  subsystem  competition  and  6  of  9  without  FSD 
subsystem  competition  18.2% 

2  or  3  of  3  with  independent  testing  and  2  or  1  of  6  without 
independent  testing  40.5% 

0  of  2  with  independent  cost  estimates  and  4  of  7  without 
independent  cost  estimates  35.7% 

4  or  3  of  7  with  low  rate  initial  production  phase  and  2  or  3 
of  3  without  low  rate  initial  production  phase  66.7% 

2  or  fewer  of  4  with  competitive  system  production  and  4  or 
more  of  6  with  non-competitive  system  production  242.9% 

2  of  6  with  competitive  subsystem  production  and  4  of  4 
with  non-competitive  subsystem  production  7.1% 

2  or  1  of  4  with  system  warranties  and  4  or  5  of  5 
without  system  warranties  40.5% 

2  of  2  with  subsystem  warranties  and  3  of  5  without  subsystem 

warranties  35.7% 

Substantial  or  Extensive  Performance  Advances  for  Probability 

7  of  8  with  AD  and  3  of  3  without  AD  72.7% 

4  of  5  with  competitive  AD  and  3  of  3  with  non-competitive  AD  62.5% 

5  of  6  with  prototypes  in  AD  and  1  of  1  without  prototypes  in  AD  85.7% 

1  of  2  with  subsystem  competition  in  FSD  and  9  of  9 

without  subsystem  competition  in  FSD  18.2% 

3  of  3  with  independent  testing  and  5  of  6  without  independent  testing  66.7% 

5  of  6  with  independent  cost  estimates  and  2  of  2  without 

independent  cost  estimates  75.0% 

6  of  7  with  low  rate  initial  production  phase  and  3  of  3 

without  low  rate  initial  production  phase  70.0% 

3  of  4  with  system  production  competition  and  6  of  6  without 

system  production  competition  40.0% 

5  of  6  with  subsystem  production  competition  and  4  of  4  without 
subsystem  production  competition  60.0% 

4  of  4  with  system  warranties  and  5  of  5  without 

subsystem  warranties  100.0% 

2  of  2  with  subsystem  warranties  and  6  of  6  without 

subsystem  warranties  100.0% 

Extensive. Materials.  Advances  for  Bcohafailitt 

3  or  fewer  of  8  with  AD  and  1  or  more  of  3  without  AD  72. 1% 

2  or  3  of  5  with  competitive  AD  and  1  or  0  of  3 

with  non-competitive  AD  7 1 .4% 

3  of  6  with  prototypes  in  AD  and  0  of  1  without  prototypes  in  AD  57.1% 

1  or  2  of  2  with  FSD  subsystem  competition  and  3  or  2  of  9 

without  FSD  subsystem  competition  89.1% 


Note-4 


2  or  3  of  3  with  independent  testing  and  1  or  0  of  6  without 
independent  testing 

3  of  6  with  independent  cost  estimates  and  0  of  2  without 
independent  cost  estimates 

3  or  4  of  7  with  low  rate  initial  production  phases  and  1  or  0  of  3 
without  low  rate  initial  production  phases 

2  or  more  of  4  with  competitive  system  production  and  2  or  fewer  of  6 
with  non-competitive  system  production 

3  or  4  of  6  with  competitive  subsystem  production  and  1  or  0  of  4 
with  non-competitive  subsystem  production 

3  or  4  of  4  with  system  warranties  and  1  or  0  of  5  without 
system  warranties 

2  of  2  with  subsystem  warranties  and  2  of  6  without  subsystem 
warranties 

Substantial  or  Extensive  Materials_Advancgs_for 
7  or  8  of  8  with  AD  and  1  or  0  of  3  without  AD 

4  of  5  with  competitive  AD  and  3  of  3  with  non-competitive  AD 

5  of  6  with  prototypes  during  AD  and  1  of  1  without  prototypes 
during  AD 

1  or  0  of  2  with  FSD  subsystem  competition  and  8  or  9  of  9 
without  FSD  subsystem  competition 

3  of  3  with  independent  testing  and  4  of  6  without  independent  testing 
5  or  6  of  6  with  independent  cost  estimates  and  1  or  0  of  2 

without  independent  cost  estimates 
5  of  7  with  low  rate  initial  production  phases  and  3  of  3 
without  low  rate  initial  production  phases 
4,  3  or  2  of  4  with  competitive  system  production  and  4,  5  or  6  of  6 
with  non-competitive  system  production 

4  of  6  with  competition  subsystem  production  and  4  of  4  with  non¬ 
competitive  subsystem  production 

3  of  4  with  system  warranties  and  5  of  5  without  system  warranties 

2  of  2  with  subsystem  warranties  and  5  of  6  without  subsystem 
warranties 

Extensive  Production  Advances  for 
2  or  fewer  of  8  with  AD  and  1  or  more  of  3  without  AD 
2  of  5  with  competitive  AD  and  0  of  3  with  non-competitive  AD 
2  of  6  with  prototypes  in  AD  and  0  of  l  without 
prototypes  in  AD 

0  of  2  with  FSD  subsystem  competition  and  3  of  9  without 
FSD  subsystem  competition 

1  or  2  of  3  with  independent  testing  and  1  or  0  of  6  without 
independent  testing 

2  of  6  with  independent  cost  estimates  and  0  of  2  without 
independent  cost  estimates 

2  or  fewer  of  7  with  low  rate  initial  production  phases  and  1  or  more 
of  3  without  low  rate  initial  production  phases 

1  or  0  of  4  with  competitive  system  production  and  2  or  3  of  6 
with  non-competitive  system  production 

2  or  3  of  6  with  competitive  subsystem  production  and  1  or  0  of  4 
with  non-competitive  subsystem  production 

2  or  3  of  4  with  system  warranties  and  1  or  0  of  5 
without  system  warranties 

2  of  2  with  subsystem  warranties  and  1  of  6  without 
subsystem  warranties 


22.6% 

35.7% 

66.7% 

£54.8% 

45.2% 

16.7% 

21.4% 

Probability 

>49.1% 

62.5% 

85.7% 

>34.5% 

41.7% 

96.4% 

46.7% 

86.7% 

33.3% 

44.4% 

75.0% 

Probability 

84.8% 

35.7% 

71  4% 

50.9% 

91.7% 

53.6% 

81.7% 

66.7% 

66.7% 

40.5% 

10.7% 


Substantial  or  Extensive  Production  Advances  for 

8  of  8  with  AD  and  1  of  3  without  AD 

5  of  5  with  competitive  AD  and  3  of  3  with  non-competitive  AD 

6  of  6  with  prototypes  in  AD  and  1  of  1  without  prototypes  in  AD 

2  of  2  with  FSD  subsystem  competition  and  7  of  9  without 
FSD  subsystem  competition 

3  of  3  with  independent  testing  and  4  of  6  without  independent 
testing 

6  of  6  with  independent  cost  estimates  and  0  of  2  without 
independent  cost  estimates 

6  or  7  or  7  with  low  rate  initial  production  phases  and  1  or  0  of  3 
without  low  rate  initial  production  phases 

4  of  4  with  competitive  system  production  and  4  of  6  with  non¬ 
competitive  system  production 

4  of  6  with  competitive  subsystem  production  and  4  of  4  with  non¬ 
competitive  subsystem  production 

4,  3  or  2  of  4  with  system  warranties  and  3, 4,  or  5  of  5 
without  system  warranties 

2  of  2  with  subsystem  warranties  and  4  of  6  without  subsystem 
warranties. 


Probability 

5.5% 

100.0% 

100.0% 

65.5% 

41.7% 

3.6% 

93.3% 

33.3% 

33.3% 

83.3% 

53.6% 


in-5.  C.f.  in-4. 
ra-6.  c.f.  in-4. 


in-7.  The  quantitative  data  provided  by  the  contractors  show  five  differences  (out  of  a 
maximum  possible  of  thirty-three)  between  the  anticipated  and  actual  levels  of 
required  technology  advance.  Two  of  the  differences  were  for  one  of  the  new 
munitions,  and  were  for  anticipated  off-the-shelf  performance  and  materials 
technology  advance  requirements  and  actual  minor  technology  advance 
requirements.  Two  more  of  the  differences  were  for  another  munition,  for 
anticipated  substantial  performance  advance  and  minor  production  advance 
requirements  and  actual  extensive  technology  advance  requirements.  The  final 
difference  was  for  anticipated  substantial  performance  advances  and  actual 
extensive  performance  advances. 


Note-6 


III-8.  The  data  for  Figures  HI-2  through  m-4  are: 


Sample 

Statistical 

Resource 

Characteristic 

Size 

Low 

High 

Mean 

Significance  Test 

Test  Equipment 

Total  Sample 

11 

15 

100 

83 

Test  Equipment 
Test  Equipment 

Intercept 

Surface  Attack 

5 

6 

67 

15 

100 

100 

85  ' 
82 

-  .662 

Mann-Whitney 

Test  Equipment 

Air  Launched 

6 

75 

100 

93  I 

-  .178 

Marm-Whitney 

Test  Equipment 

Surface  Launched 

5 

15 

100 

71  - 

Test  Equipment 

Army 

5 

15 

100 

71  1 

Test  Equipment 

Navy 

4 

75 

100 

94 

-  >.100 

Kruskal- Wallis 

Test  Equipment 

Air  Force 

2 

90 

95 

93  J 

Test  Equipment 
Test  Equipment 

New 

Modified 

7 

4 

85 

15 

100 

90 

96] 

62  J 

-  .018 

Marm-Whitney 

Facilities 

Total  Sample 

11 

0 

100 

50 

Facilities 

Facilities 

Intercept 

Surface  Attack 

5 

6 

10 

0 

30 

100 

17  * 
78 

|-  .082 

Mann-Whitney 

Facilities 

Facilities 

Air  Launched 
Surface  Launched 

6 

5 

10 

0 

100 

100 

53  * 
47 

-  .428 

Marm-Whitney 

Facilities 

Army 

5 

0 

100 

47  1 

Facilities 

Navy 

4 

10 

100 

56 

-  >.100 

Kruskal-Wallis 

Facilities 

Air  Face 

2 

10 

80 

45  J 

Facilities 

New 

7 

10 

100 

62  1 

L  .230 

Marm-Whitney 

Facilities 

Modified 

4 

0 

80 

29 

Tooling 

Total  Sample 

11 

0 

100 

75 

Tooling 

Tooling 

Intercept 

Surface  Attack 

5 

6 

0 

15 

100 

100 

66  -i 

83  j 

|»  .482 

Mann-Whitney 

Tooling 

Tooling 

Air  Launched 
Surface  Launched 

6 

5 

0 

15 

100 

100 

80 

69  _ 

|_  .599 

Mann-Whitney 

Tooling 

Army 

5 

15 

100 

69  "1 

Tooling 

Navy 

4 

0 

100 

75 

-  >.100 

Kruskal-Wallis 

Tooling 

Air  Force 

2 

80 

100 

90  _ 

Tooling 

Tooling 

New 

Modified 

7 

4 

80 

0 

100 

80 

97  I 
36  J 

-  .009 

Marm-Whitney 

IH-9.  Spearman  rank  correlations  between  resource  requirements  and  development  start 
dates  are  shown  below: 


Resources 

Correlation 

Coefficient 

Significance 

Level 

Test  Equipment  Requirements 

-.293 

33.1% 

Facilities  Requirements 

-.173 

56.7% 

Tooling  Requirements 

.000 

100.0% 

HI- 10.  Spearman  rank  correlations  between  the  three  categories  of  resource  requirements 
are  shown  below: 


Resources 

Correlation 

Coefficient 

Significance 

Level 

Test  Equipment  Requirements 

Facilities  Requirements 

+.516 

8.7% 

Test  Equipment  Requirements 

Tooling  Requirements 

+.736 

1.5% 

Facilities  Requirements 

Tooling  Requirements 

+.445 

14.0% 

HI- 1 1 .  The  data  for  Figure  HI-5  are: 


Advance 

Sample 

Statistical 

Resource 

Technology 

Level 

Size 

Low 

High 

Mean 

Significance 

Test 

Test  Equipment 

Performance 

Mine* 

1 

100 

100 

100 

Test  Equipment 

Performance 

Substantial 

4 

15 

100 

70 

-  .405 

Kruskal-Wallis 

Test  Equipment 

Performance 

Extensive 

6 

67 

100 

90  , 

Test  Equipment 

Materials 

Off-the-Shelf 

1 

75 

75 

75" 

Test  Equipment 

Materials 

Minor 

1 

100 

100 

100 

-  .500 

Kruskal-Wallis 

Test  Equipment 

Materials 

Substantial 

5 

15 

100 

80 

Test  Equipment 

Materials 

Extensive 

4 

67 

100 

86 

Test  Equipment 

Production 

Minor 

2 

15 

75 

451 

l-  .072 

Mann-Whiteny 

Test  Equipment 

Production 

Substantial 

6 

90 

100 

96  J 

I^J_  .321 

Mann-Whitney 

Test  Equipment 

Production 

Extensive 

3 

67 

100 

84 

Facilities 

Peformance 

Minor 

1 

100 

100 

100" 

Facilities 

Performance 

Substantial 

4 

0 

100 

49 

-  .440 

Kruskal-Wallis 

Facilities 

Performance 

Extensive 

6 

10 

100 

43  J 

Facilities 

Materials 

Off-the-Shelf 

1 

15 

15 

151 

Faciliteis 

Materials 

Minor 

1 

100 

100 

100 

-  .274 

Kruskal-Wallis 

Facilities 

Materials 

Substantial 

5 

0 

100 

30 

Facilities 

Materials 

Extensive 

4 

20 

100 

71 J 

Facilities 

Production 

Minor 

2 

0 

15 

81 

Facilities 

Production 

Substantial 

6 

10 

100 

55 

■  .211 

Kruskal-Wallis 

Facilities 

Production 

Extensive 

3 

20 

100 

68 

Tooling 

Performance 

Minor 

1 

100 

100 

100 " 

Tooling 

Performance 

Substantial 

4 

0 

100 

49 

.227 

Kruskal-Wallis 

Tooling 

Performance 

Extensive 

6 

50 

100 

88 . 

Tooling 

Mateirals 

Off-the-Shelf 

1 

0 

0 

0 

Tooling 

Materials 

Minor 

1 

100 

100 

100 

-  .322 

Kruskal-Wallis 

Tooling 

Materials 

Substantial 

5 

15 

100 

79 

Tooling 

Materials 

Extensive 

4 

50 

100 

83  J 

Tooling 

Production 

Minor 

2 

0 

15 

8) 

•  072 

Mann-Whitney 

Tooling 

Production 

Substantial 

6 

80 

100 

93  J 
83 

J-  .904 

Mann-Whitney 

Tooling 

Production 

Extensive 

3 

50 

100 

Note-8 


12.  The  data  for  Figure  HI-6  are  shown  below.  All  of  the  statistical  significance  tests 
are  Mann-Whitney,  interpreted  using  the  Tables  of  Probabilities  and  Critical 
Values  in  the  Chemical  Rubber  Corporation  Handbook  of  Tables  for  Probability 
and  Statistics  for  small,  unequal  sample  sizes. 


Resource 

Initiative 

Sample 

Size 

Low  High 

Mean 

Statistical 

Significance 

Test  Equipment  With  Advanced  Development  Phase 

8 

85 

100 

95 

\  fin 

Test  Equipment  Without  Advanced  Development  Phase 

3 

15 

75 

52 

Facilities 

With  Advanced  Development  Phase 

8 

10 

100 

64 

>  1 XA 

Facilities 

Without  Advanced  Development  Phase 

3 

0 

20 

10 

Tooing 

With  Advanced  Development  Phase 

8 

80 

100 

95 

\  nn 

Tooling 

Without  Advanced  Development  Phase 

3 

0 

50 

22 

Test  Equipment 

With  Advanced  Development  Prototypes 

6 

85 

100 

95 

\  71 A 

Test  Equipment  Without  Advanced  Development  Prototypes 

1 

100 

100 

100 

Facilities 

With  Advanced  Development  Prototypes 

6 

10 

100 

64 

►  577 

Facilities 

Without  Advanced  Develpment  Prototypes 

1 

100 

100 

100 

Tooling 

With  Advanced  Development  Prototypes 

6 

80 

100 

97 

\  i  nnn 

Tooling 

Without  Advanced  Development  Prototypes 

1 

100 

100 

100 

I  1  .wu 

Test  Equipment 

With  FSD  Subsystem  Competition 

2 

90 

100 

95 

l  w  7nn 

Test  Equipment 

Without  FSD  Subsystem  Competition 

9 

15 

100 

81 

J  >.zuu 

Facilities 

With  FSD  Subsystem  Competition 

2 

80 

100 

90 

l  ^  7nn 

Facilities 

Without  FSD  Subsystem  Competition 

9 

0 

100 

41 

j  >.zuu 

Tooling 

With  FSD  Subsystem  Competition 

2 

80 

100 

90 

l  ^  700 

Tooling 

Without  FSD  Subsystem  Competition 

9 

0 

100 

72 

r  >.zw 

Test  Equipment 

With  Independent  Cost  Estimate 

6 

& 

100 

95 

l  077 

Test  Equipment 

Without  Independent  Cost  Estimate 

2 

15 

75 

45 

j  .u/Z 

Facilities 

With  Independent  Cost  Estimate 

6 

10 

100 

79 

Facilities 

Without  Independent  Cost  Estimate 

2 

0 

15 

8 

J  .IHZ 

Tooling 

With  Independent  Cost  Estimate 

6 

80 

100 

97 

\  077 

Tooling 

Without  Independent  Cost  Estimate 

2 

0 

15 

8 

Test  Equipment 

With  Low  Rate  Initial  Production  Phase 

'7'  ' 

"ii' 

100 

92 

Test  Equipment  Without  Low  Rate  Initial  Production  Phase 

3 

15 

90 

57 

J  .U!#l 

Facilities 

With  Low  Rate  Initial  Production  Phase 

7 

10 

100 

57 

Facilities 

Without  Low  Rate  Initial  Production  Phase 

3 

0 

30 

17 

j  .J<$4 

Tooling 

With  Low  Rate  Initial  Production  Phase 

7 

0 

100 

83 

Tooling 

Without  Low  Rate  Initial  Production  Phase 

3 

15 

80 

48 

J 

Test  Equipment  With  System  Production  Competition 

4 

67 

100 

88 

i  Q7ff 

Test  Equipment 

Without  System  Production  Competition 

6 

15 

100 

78 

Facilities 

With  System  Production  Competition 

4 

10 

100 

53 

Facilities 

Without  System  Production  Competition 

6 

0 

100 

40 

f  . /oz 

Tooling 

With  System  Production  Competition 

4 

50 

100 

83 

\  a^a 

Tooling 

Without  System  Production  Competition 

6 

0 

100 

66 

Test  Equipment 

With  Subsystem  Production  Competition 

6 

15 

100 

75 

\  i 

Test  Equipment 

Without  Subsystem  Production  Competition 

4 

85 

100 

93 

Facilities 

With  Subystem  Production  Competition 

6 

0 

100 

53 

1  i  nnn 

Facilities 

Without  Subsystem  Production  Competition 

4 

10 

85 

34 

Tooling 

With  Subsystem  Production  Competition 

6 

0 

100 

58 

i  715 

Tooling 

Without  Subystem  Production  Competition 

4 

80 

100 

95 

;  .zis 

Test  Equipment 

With  System  Warranty 

4 

75 

100 

88 

r  004 

Test  Equipment 

Without  System  Warranty 

5 

15 

100 

73 

Test  Equipment  With  Subsystem  Warranty 

"‘2"" 

"6 T 

"i'00‘ 

'"84" 

\  i  non 

Test  Equipment 

Without  Subsystem  Warranty 

6 

15 

100 

76 

HI-13.  The  quantitative  data  provided  by  the  contractors  show  six  differences  (out  of  a 
maximum  possible  of  thirty-three)  for  four  separate  munitions,  between  the 
anticipated  and  actual  requirements  for  the  three  types  of  resources.  Three  of  the 
differences  were  small,  for  three  different  munitions:  one  overestimate  of 
percentage  requirement  for  new  test  equipment  by  5  percent  (100  percent 
anticipated  versus  95  percent  actual);  and  two  underestimates  of  the  percentage 
requirements  for  new  facilities  by  5  percent  (10  percent  anticipated  versus  15 
percent  actual,  and  25  percent  anticipated  versus  30  percent  actual).  The  other 
three  differences  are  for  large  underestimates  for  each  of  the  three  resources  for 
one  munition:  a  40-percent  underestimate  of  the  requirement  for  new  test 
equipment  (50  percent  anticipated  versus  90  percent  actual);  and  50-percent 
underestimates  of  the  percentage  requirements  for  both  new  facilities  and  new 
tooling  (30  percent  anticipated  versus  80  percent  actual). 

HI- 1 4.  Spearman  rank  correlations  between  the  three  categories  of  resource  requirements 
anticipation  error  are  shown  below: 


Resource  Requirements 

Correlation 

Significance 

Anticipation  Errors 

Coefficient 

Level 

>10.0% 

>10.0% 

>10.0% 


HI- 15.  A  measure  of  development  macro-stability  was  also  calculated  for  each  munition 
in  the  sample,  but  those  measures  were  found  to  have  no  relationship  to  any  of 
the  development  outcome  measures  evaluated  in  Chapter  IV  nor  to  any  of  the 
other  risk  measures  discussed  in  Chapter  in.  The  measure  of  development 
macro-stability  for  each  munition  was  calculated  as  the  compound  annual  growth 
for  defense  research,  development,  test  and  evaluation  total  obligational  authority, 
from  the  start  of  full  scale  development  to  initial  operational  capability.  The  data 
are  shown  below: 


Designator 

Title 

Compound  Annual 
Growth  Rate 

A/RIM-7E 

Sparrow  III  B  CW 

4.49 

AIM-7F 

Sparrow  III  Pulse  Doppler 

-2.60 

A/RIM-7M 

Sparrow  III  Monopulse 

5.55 

AIM-9L 

Sidewinder 

-0.64 

AIM-9M 

Sidewinder 

3.78 

AIM-54A 

Phoenix 

-3.93 

AIM-54C 

Phoenix 

6.15 

AIM- 120  A 

AMRAAM 

5.20 

F1M-92A 

Stinger-Basic 

0.34 

FIM-92A 

Stinger- POST/RMP 

5.60 

AGM-65D/F/G 

HR  Maverick 

8.30 

A/R/UGM-84  A/C/D 

Harpoon 

-1.34 

AGM-88A 

HARM 

6.65 

AGM-114A/B 

Hellfire 

6.44 

BOM-71  A 

TOW  I 

-2.86 

BGM-71D 

Town 

9.83 

- 

MLRS 

5.55 

M-712 

Copperhead  CLGP 

4.80 

- 

5"  Deadeye  SALGP 

3.78 

Number  of  Observations 

19 

Note- 10 


HI- 16.  For  those  years  that  the  budgeted  amount  was  zero,  no  error  was  calculated  Nor 
were  any  errors  calculated  for  those  years  for  which  either  the  budgeted  amount 
or  appropriated  amount  could  not  be  found  in  the  SARs.  In  the  calculation,  the 
errors  are  squared,  to  keep  the  positive  and  negative  errors  from  offsetting  one 
another.  The  squared  errors  for  a  munition  were  averaged  and  the  square  root  of 
the  average  was  calculated.  Use  of  this  measure  weights  (for  example)  a  50- 
percent  cut  in  a  small  budget  the  same  as  a  50-percent  cut  in  a  large  budget  An 
alternative  micro-measure  could  be  calculated  as  the  root  mean  squared  annual 
error,  normalized  by  the  mean  annual  budget;  use  of  this  latter  measure  would 
give  a  50-percent  cut  in  a  small  budget  much  less  impact  than  a  50-percent  cut  in  a 
large  budget  The  former  measure  was  used  because  it  was  believed  that  the 
annual  percentage  error  was  more  important  to  the  contractor  in  planning  and 
managing  his  program. 

m-17.  The  data  for  Figure  HI-7  are  shown  below.  All  of  the  statistical  tests  are  Kruskal- 
Wallis. 


Sample 

Statistical 

Category 

Size 

Low 

High 

Mean 

Significance 

Total  Sample 

18 

-3.99 

9.79 

1.81 

Intercept 

10 

-3.99 

9.79 

1.85  ' 

j>  .789 

Surface  Attack 

8 

-3.37 

9.51 

2.22  . 

Air  Launched 

12 

-3.99 

8.49 

1.20  ' 

}  .373 

Surface  Launched 

6 

-2.34 

9.79 

3.01  ■ 

Army 

7 

-2.34 

9.79 

2.56  1 

Air  Force 

2 

-3.99 

-3.37 

-3.68 

-  .056 

Navy 

9 

-3.98 

8.49 

2.44  J 

New 

9 

-3.99 

9.79 

2.51  1 

\  .479 

Modified 

9 

-3.98 

8.49 

1.10  J 

ffl-18.  The  data  for  Figure  HI-8  are  shown  below.  All  of  the  statistical  tests  are  Mann- 
Whitney,  interpreted  using  the  Table  of  Probabilities  in  the  Chemical  Rubber 
Corporation  Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal 
sample  sizes. 


Category 

Sample 

Size 

Low 

High 

Mean 

Statistical 

Significance 

Total  Sample 

11 

.073 

.608 

.262 

Intercept 

5 

.073 

.479 

.208  \ 

.482 

Surface  Attack 

6 

.092 

.608 

.307  •> 

Air  Launched 

8 

.073 

.508 

.240  \ 

.702 

Surface  Launched 

3 

.119 

.608 

.321  J 

Army 

4 

.092 

.608 

.264  \ 

^  1.000 

Navy 

5 

.073 

.276 

.167  •>' 

Air  Force 

2 

.479 

.508 

.494  • 

i  .094 

New 

6 

.092 

.608 

.282  1 

1.000 

Modified 

5 

.073 

.508 

.237  J 

III-19.  Spearman  rank  correlations  between  production  start  dates  and  funding  stability 
measures  are  shown  below: 


Funding  Stability  Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Procurement  TOA  Growth  Rate 

18 

-.431 

7.5% 

Root  Mean  Squared  Annual  Percentage 

11 

+.141 

65.5% 

Difference  Between  Program  Procure- 

ment  Requests  and  Appropriations 

ni-20.  Spearman  rank  correlations  between  stability  measures  and  requirements  for  new 
resources  are  shown  below: 


Measures 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

• 

Root  Mean  Squared  Annual  Percentage 
Difference  Between  Program  Procure¬ 
ment  Requests  and  Appropriations 
Procurement  TOA  Annual  Growth  Rate 

11 

+.086 

78.5% 

New  Test  Equipment 

10 

-.082 

80.6% 

New  Facilities 

10 

+.330 

32.2% 

• 

New  Tooling 

10 

.000 

100.0% 

Root  Mean  Squared  Annual  Percentage  Difference 
Between  Program  Procurement  Requests  and 
Appropriations 

New  Test  Equipment 

8 

-.137 

>10.0% 

New  Facilities 

8 

-.012 

>10.0% 

A 

New  Tooling 

8 

+.250 

>10.0% 

The  significance  levels  for  the  sample  sizes  of  8  were  interpreted  using  the  Table 
of  Critical  Values  from  the  Chemical  Rubber  Company  Handbook  of  Tables  for 
Probability  and  Statistics. 

m-2 1 .  The  data  for  Figure  HI-9  are  shown  below.  All  of  the  statistical  significance  tests 
are  Kruskal- Wallis. 


Advance 

Sample 

Statistical 

Technology 

Level 

Size 

Low 

High 

Mean 

Significance 

Performance 

Minor 

1 

.092 

.092 

.092] 

Performance 

Substantial 

3 

.122 

.508 

.289 

-  >.129 

Performance 

Extensive 

4 

.119 

.608 

•37  lj 

Materials 

Off-the-Shelf 

1 

.122 

.122 

.122] 

Materials 

Minor 

1 

.092 

.092 

.092 

-  >.100 

Materials 

Substantial 

3 

.119 

.479 

.278 

Materials 

Extensive 

3 

.276 

.608 

.464  J 

Production 

Minor 

2 

.122 

.236 

.179* 

Production 

Substantial 

4 

.092 

.508 

.300 

■  >.102 

Production 

Extensive 

2 

.276 

.608 

,442j 

Note- 12 


m-22.  The  data  for  Figure  HI- 10  are  shown  below: 


Sample  Statistical 

Initiative  Size  Low  High  Mean  Significance  Test 


With  Limited  Initial  Production  Phase  7 
Without  Limited  Initial  Production  Phase  1 


With  Competitive  System  Production  5 

Without  Competitive  System  Production  6 
With  Competitive  Subsystem  Production  7 
Without  Competitive  Subsystem  Production  4 
With  Multi-year  Production  Contracting  2 
Without  Multi-year  Production  Contracting  9 
With  Foreign  Production  3 

Without  Foreign  Production  8 


.092 

.608 

.315  I 

.119 

.119 

.119  J 

■  .375 

Mann-Whitney 

.073 

.508 

.280  I 

.122 

.608 

.247  J 

r  .930 

Mann-Whitney 

.073 

.508 

.222  I 

-  .648 

.119 

.608 

.331  J 

Mann-Whitney 

.236 

.608 

.422-, 

.073 

.508 

.226  J 

►  >.200 

Mann-Whitney 

.119 

.508 

.369  ", 

.073 

.608 

.256] 

.  .630 

Mann-Whitney 

HI-23.  The  development/production  overlap  ratio  calculation  data  for  Table  HI-5  are 
shown  below.  The  IOC,  MS  n,  and  MS  HI  dates  are  from  Table  II- 1. 


Designator 

Title 

IOC 

msh 

Msm 

ioc-ms  m 

ioc-ms  n 

A/RIM-7E 

Sparrow  M-B  CW 

12/63 

1/60 

I.U. 

I.U. 

47 

AIM-7F 

Sparrow  III  Pulse  Doppler 

4.76 

12/65 

I.U. 

I.U. 

124 

A/RIM-7M 

Sparrow  m  Monopulse 

1/83 

4/78 

11/82 

2 

54 

AIM-9L 

Sidewinder 

5/78 

8/71 

1/76 

28 

81 

AIM-9M 

Sidewinder 

9/82 

2/76 

6/81 

19 

79 

AIM-54A 

Phoenix 

12/73 

I.U. 

I.U. 

I.U. 

I.U. 

AIM-54C 

Phoenix 

12/86 

10/76 

12/79 

84 

122 

AIM-120A 

AMRAAM 

10/89 

11/82 

3/89 

6 

83 

FIM-92A 

Stinger 

2/81 

5/72 

11/77 

39 

105 

FIM-92A 

Stinger-POST/RMP 

I.U. 

I.U. 

I.U. 

I.U. 

I.U. 

AGM-65D/F/G 

HR  Maverick 

2/86 

9/76 

9/82 

41 

120 

A/R/UGM-84  A/C/D 

Harpoon 

7/77 

3/74 

6/75 

25 

40 

AGM-88A/B 

HARM 

11/83 

2/78 

3/83 

8 

69 

AGM-114A/B 

Hellfire 

7/86 

2/76 

3/82 

52 

125 

BGM-71A 

TOW  I 

9/70 

I.U. 

I.U. 

I.U. 

I.U. 

BGM-71D 

Town 

10/83 

9/78 

9/81 

25 

61 

- 

MLRS 

3/83 

N.A. 

5/80 

34 

N.A. 

M-712 

Copperhead  CLGP 

12/82 

6/75 

11/79 

37 

70 

- 

5"  Deadeye  SALGP 

N.A. 

11/77 

N.A. 

N.A. 

N.A. 

I.U.  -  Information  unavailable. 
N.A.  -  Not  applicable. 


The  development/production  overlap  ratios  in  Table  m-5  are  highly  correlated 
with  the  numbers  of  months  between  MS  III  and  IOC;  the  Spearman  rank 
correlation  is  .738  with  a  statistical  significance  of  01.4%,  which  means  that  a 
high  development/production  overlap  ratio  is  indicative  of  a  larger  number  of 
months  of  overlap,  but  not  of  a  shorter  development  schedule. 


ni-24.  The  data  for  Figure  Hl-11  are  shown  below.  The  Mann- Whitney  tests  are 
interpreted  using  the  Tables  of  Probabilities  and  Critical  Values  in  the  Chemical 
Rubber  Corporation  Handbook  of  Tables  for  Probability  and  Statistics  for  small, 
unequal  sample  sizes. 


Sample 

Statistical 

Characteristic 

Size 

Low 

High 

Mean 

Significance 

Test 

Total  Sample 

12 

.027 

.689 

.350 

Intercept 

Surface  Attack 

6 

6 

.037 

.116 

.689 

.625 

.293 

.406 

}  .310 

Mann- Whitney 

Air  Launched 

10 

.037 

.689 

.326 

}  >.200 

Mann- Whitney 

Surface  Launched 

2 

.410 

.529 

.470 

Army 

4 

.371 

.529 

.432  ' 

1 

Navy 

6 

.037 

.689 

.342 

l  264 

Kruskall-Wailis 

Air  Force 

2 

.072 

.342 

.207  _ 

\ 

New 

6 

.072 

.625 

.355  ' 

f  .700 

Mann- Whitney 

Modified 

6 

.037 

.689 

.344  . 

El-25.  Spearman  rank  correlations  between  the  development/production  overlap  ratios 
and  production  start  dates  and  funding  stability  measures  are  shown  below: 


Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Production  Start  Date 

12 

-.580 

5.4% 

Procurement  70A  Annual  Growth  Rate 

Root  Mean  Squared  Annual  Percentage  Difference 
Between  Program  Procurement  Requests 

12 

+.525 

8.2% 

and  Appropriations 

11 

-.059 

85.2% 

m-26.  Spearman  rank  correlations  between  the  development/production  overlap  ratios 
and  the  percentage  requirements  for  new  resources  are  shown  below: 


Resource  Category 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Requirements  for  New  Test  Equipment 

8 

-.512 

>10.0% 

Requirements  for  New  Facilities 

8 

+.048 

>10.0% 

Requirements  for  New  Tooling 

8 

-.238 

>10.0% 

The  significance  levels  were  interpreted  using  the  Table  of  Critical  Values  from 
the  Chemical  Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics 
for  small  sample  sizes. 


m-27.  The  data  for  Figure  III- 12  are  shown  below.  All  of  the  statistical  tests  are 
Kruskal- Wallis. 


Advance 

Sample 

Statistical 

Technology 

Level 

Size 

Low 

High 

Mean 

Significance 

Test 

Performance 

Minor 

1 

.416 

.416 

.416  I 

Performance 

Substantial 

3 

.342 

.689 

.480 

-  >.129 

Kruskal- Wallis 

Performance 

Extensive 

4 

.072 

.529 

.272  J 

Materials 

Off-the-Shelf 

1 

.689 

.689 

.689  I 

Materials 

Materials 

Minor 

Substantial 

1 

3 

.416 

.072 

.416 

.410 

.416 

.284 

.  .359 

Kruskal -Wallis 

Materials 

Extensive 

3 

.116 

.529 

.329  J 

Production 

Minor 

2 

.410 

.689 

.550  1 

Production 

Substantial 

4 

.072 

.416 

.300 

*  >.105 

Kruskal- W allis 

Production 

Extensive 

2 

.116 

.529 

.323  J 

HI-28.  The  data  for  Figure  HI- 13  are  shown  below.  All  of  the  statistical  significance 
tests  are  Mann- Whitney,  interpreted  using  the  Table  of  Probabilities  in  the 
Chemical  Rubber  Corporation  Handbook  of  Tables  for  Probability  and  Statistics 
for  small,  unequal  sample  sizes. 


Statis- 

deal 

Sample 

Signifi- 

Initiatives 

Size 

Low 

High  Mean 

cance  Test 

With  Independent  Testing 

Without  Independent  Testing 

2 

9 

.116 

.037 

.342 

.689 

.229  . 
.374  _ 

J.>.200  Marm-Whitney 

With  Independent  Cost  Estimate 

5 

.072 

.529 

.295  . 

J,  .662  Mann-Whitney 

Without  Independent  Cost  Estimate 

6 

.037 

.689 

.391 

With  Low-Rate  Initial  Production  Phase 
Without  Low-Rate  Initial  Production  Phase 

7 

3 

.072 

.346 

.689 

.410 

.398 

.376 

j.  .834  Mann-Whitney 

With  System  Production  Competiton 

6 

.037 

.416 

.242 

j.  .064  Mann-Whitney 

Without  System  Competition 

6 

.116 

.625 

.457 

IV- 1.  The  statistical  significance  of  the  relationships  between  munition  category  and 
whether  or  not  the  munition  satisfied  the  operational  and  technical  requirements 
was  calculated  from  Fisher's  exact  probability  test  for  a  2x2  contingency  table 
with  small  sample  sizes: 


Operational  and  Technical  Requirements  Satisfied  bv  Probability 

3  or  fewer  of  8  intercept  and  4  or  more  of  9  surface  attack  38.1% 

3  or  fewer  of  1 1  air-launched  and  4  or  rim  of  6  surface-launched  14.3% 

3  or  fewer  of  9  new  and  4  or  more  of  8  modified  41.9% 


IV-2. 


A  test  using  the  Table  of  the  Total  Number-of-Runs  Test  few  Randomness  in  the 
Chemical  Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for 
small,  unequal  sample  sizes  was  performed  to  test  whether  or  not  satisfaction  of 
operational  and  technical  requirements  is  randomly  distributed  over  time.  There 
are  nine  runs  in  the  data.  The  probability  of  nine  or  fewer  runs  with  samples  of 
seven  munitions  satisfying  the  requirements  and  10  munitions  not  satisfying  the 
requirements  is  equal  to  .549. 

IV-3.  The  statistical  significance  of  the  relationships  between  whether  or  not  an 
acquisition  policy  was  applied  and  whether  or  not  the  munition  satisfied  the 
operational  and  technical  requirements  was  calculated  from  Fisher's  exact 
probability  test  for  a  2x2  contingency  table  with  small  sample  sizes: 


Operational  and  Technical  Requirements  Satisfied  bv  EmhabilitS 

4  or  fewer  of  7  with  AD  and  7  or  more  of  10  without  AD  85.5% 

0  of  4  with  competitive  AD  and  5  of  7  with  non-competitive  AD  19.7% 

2  or  fewer  of  6  with  prototypes  in  AD  and  7  or  more  of  10 

without  prototypes  in  AD  18.2% 

1  or  0  of  7  with  FSD  subsystem  competition  and  3  or  4  of  9 
without  FSD  subsystem  competition  39.2% 

1  or  0  of  6  with  independent  testing  and  2  or  3  of  10 
without  independent  testing  69.6% 

3  or  more  of  6  with  independent  cost  estimates  and  2  or  fewer  of  9 

without  independent  cost  estimates  28.7% 

3  or  fewer  of  6  with  low-rate  initial  production  phases  and  5  or  more  of 
8  without  low-rate  initial  production  phases  52.9% 


IV-4.  The  statistical  significance  of  the  relationships  between  technology  advance  and 
whether  or  not  the  munition  satisfied  the  operational  and  technical  requirements 
was  calculated  from  Fisher’s  exact  probability  test  for  a  2x2  contingency  table 
with  small  sample  sizes: 

Operational  and  Technical  Requirements  Satisfied  to  Probability 

2  or  1  of  5  with  extensive  advances  and  4  or  5  with  substantial 
or  lesser  advances  in  performance  technology  40.5% 

2  or  fewer  of  3  with  extensive  advances  and  4  or  more  of  6 
with  substantial  or  lesser  advances  in  materials  technology  77.4% 

1  or  0  of  2  with  extensive  advances  and  5  or  6  of  7 
with  substantial  or  lesser  advances  in  production  technology  91.7% 

IV-5.  The  data  for  Figure  IV- 1  are  shown  below.  The  Mann-Whitney  tests  were 
interpreted  using  the  Table  of  Probabilities  from  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal  sample 
sizes. 


Requirements 

Sample 

Statistical 

Resource 

Satisfied 

Size 

Low 

High 

Mean 

Significance  Test 

Test  Equipment 
Test  Equipment 

Yes 

No 

6 

3 

15 

100 

100 

100 

76  ] 
100  ‘ 

\  .072 

Mann-Whitney 

Facilities 

Facilities 

Yes 

No 

6 

3 

0 

10 

100 

100 

52  1 
70  J 

\  .548 

Mann-Whitney 

Tooling 

Yes 

6 

0 

100 

63  1 

\  .166 

Mann-Whitney 

Tooling 

No 

3 

100 

100 

100  J 

IV-6.  The  data  for  Figure  IV- 2  are  shown  below: 


Requirement 

Sample 

Statistical 

Satisfied 

Size 

Low 

High 

Mean 

Significance 

Test 

Yes 

S 

.342 

.689 

.468 

}  .246 

Mann-Whitney 

No 

6 

.037 

.625 

.297 

IV-7.  The  data  for  Figure  IV- 3  are  shown  below: 


Measure 

Characteristic 

Sample 

Size 

Low 

High 

Mean 

Advanced  Development 

All- Actual 

9 

17 

75 

42 

Full  Scale  Development 

All-Planned 

19 

33 

127 

57 

Full  Scale  Development 

Alt- Actual 

19 

47 

134 

85 

Full  Scale  Development 

Intercept-Planned 

10 

33 

127 

62 

Full  Scale  Development 

Intercept-Actual 

10 

47 

134 

90 

Full  Scale  Development 

Surface  Attack-Planned 

9 

37 

87 

59 

Full  Scale  Development 

Surface  Attack-Actual 

9 

49 

125 

80 

Full  Scale  Development 

Air  Launched-Planned 

12 

33 

87 

55 

Full  Scale  Development 

Air  Launched- Actual 

12 

47 

125 

85 

Full  Scale  Development 

Surface  Launched-Planned 

7 

49 

127 

69 

Full  Scale  Development 

Surface  Launched- Actual 

7 

49 

134 

85 

Full  Scale  Development 

Army-Planned 

7 

52 

127 

74 

Full  Scale  Development 

Army- Actual 

7 

61 

134 

96 

Full  Scale  Development 

Navy-Planned 

10 

33 

84 

52 

Full  Scale  Development 

Navy-Actual 

10 

47 

124 

73 

Full  Scale  Development 

Air  Force-Planned 

2 

54 

57 

56 

Full  Scale  Development 

Air  Force-Actual 

2 

94 

113 

104 

Full  Scale  Development 

New-Planned 

10 

37 

87 

58 

Full  Scale  Development 

New-Actual 

10 

49 

125 

80 

Full  Scale  Development 

Modified-Pl  aimed 

9 

33 

127 

63 

Full  Scale  Development 

Modified- Actual 

9 

47 

134 

91 

IV-8.  The  data  for  Figure  IV -4  are  shown  below.  The  Mann- Whitney  test  was 
interpreted  using  the  Table  of  Probabilities  in  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal  sample 
sizes. 


Sample  Statistical 


Characteristic 

Size 

Low 

High 

Mean  Significance  Test 

Total  Sample 

19 

1.00 

2.82 

1.48 

Intercept 

10 

1.00 

2.82 

1.58  1 

.462 

Kruakai-Wallii 

• 

Surface  Attack 

9 

1.00 

1.98 

136  J 

Air  Launched 

Surface  Launched 

12 

7 

1.00 

1.00 

2.82 

1.73 

139  1 

128  l 

.271 

Kiuskal-Wallis 

Navy 

!0 

1.00 

2.82 

1.49  1 

.845 

Kiuskal-Wallis 

Army 

Air  Force 

7 

2 

1.00 

1.74 

1.73 

1.98 

1.34  J  ' 
1.86  . 

|  .056 

Mann- Whitney 

• 

New 

10 

1.00 

1.74 

'll) 

.744 

Kiuskal-Wallis 

Modified 

9 

1.00 

2.82 

1.58  J 

Operational/Technical 
Requirements  Satisfied 
Opeiatiooal/Technical 

7 

1.00 

1.98 

1.40  1 

.845 

Kiuskal-Wallis 

• 

Requirements  not  Satisfied 

10 

1.00 

2.82 

1.54  J 

IV-9.  The  data  for  Figure  IV- 5  are  shown  below.  The  Mann- Whitney  tests  were 
interpreted  using  the  Tables  of  Probabilities  and  Critical  Values  in  the  Chemical 
Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for  small,  • 

unequal  sample  sizes. 


Statis¬ 

tical 

Sample  Signifi- 


Initiative 

Size 

Low 

High  Mean 

canoe 

Test 

A 

With  Advanced  Development  Phase 

12 

1.00 

1.98  1.44  1 

•  .672 

Kiuskal-Wallis 

• 

Without  Advanced  Development  fhase 

7 

1.00 

2.82  134  J 

With  Advanced  Development  Prototypes 

10 

1.00 

1.98  1.46  1 

•  .327 

Kiuskal-Wallis 

Without  Advanced  Development  Prototypes 

8 

1.00 

2.82  1.48  J 

With  Competitive  Advanced  Development 

With  Non-Competitive  Advanced  Development 

7 

5 

1.00 

1.00 

1.74  1.44i 
1.98  1.43  J 

■  .816 

Mann-Whitney 

• 

With  FSD  Subsystem  Competition 

Without  FSD  Subsystem  Competition 

4 

14 

1.35 

1. 00 

1.98  1.57  1 
2.82  1.48  > 

•  >.200 

Mann-Whitney 

With  Independent  Testing 

Without  Independent  Testing 

3 

14 

1.19 

1.00 

1.98  1.46  1 
2.82  1.50  J 

8 

C9 

A 

Mann-Whitney 

With  Independent  Cost  Estimate 

6 

1.00 

1.98  1.521 

•  .664 

Kruskal-Wallis 

• 

Without  Independent  Cost  Estimate 

10 

1.00 

2.82  1.51  > 

With  System  Warranties 

6 

1.21 

2.82  1.78  1 

•  .039 

Kruskal-Wallis 

Without  System  Warranties 

10 

1.00 

1.74  1.25  J 

With  Subystem  Warranties 

Without  Subsystem  Warranties 

3 

12 

1.01 

1.00 

1.21  1.09  l 
2.82  1.51  J 

•  >.200 

Mann-Whitney 

• 

Note-18 


IV- 10.  The  data  for  Figure  IV- 6  are  shown  below.  The  Mann- Whitney  test  was 
interpreted  using  the  Table  of  Critical  Values  in  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal  sample 
sizes. 


Sample 

Statistical 

Risk  Measure 

Size 

Low 

High 

Mean 

Significance 

Test 

Stable  Threat 

Unstable  Threat 

17 

2 

1.00 

1.21 

2.82 

1.98 

1.46  1 
1.60  J 

\  >.200 

Mann-Whitney 

Technology  Advance  Subsample 
Remainder  of  Sample 

11 

8 

1.00 

1.00 

1.98 

2.82 

1.41  1 
1.58  J 

\  .934 

Kruskal -Wallis 

Minor  Performance  Advance 

1 

1.44 

1.44 

1.44  I 

Substantial  Performance  Advance 

4 

1.00 

1.98 

1.36 

-  .843 

Kruskal-Wallis 

Extensive  Performance  Advance 

6 

1.06 

1.74 

1.43  J 

No  Materials  Advance 

1 

1.45 

1.45 

1.45  ‘ 

Minor  Materials  Advance 

1 

1.44 

1.44 

1.44 

-  .823 

Kruskal- Wallis 

Substantial  Materials  Advance 

5 

1.00 

1.74 

1.32 

Extensive  Materials  Advance 

4 

1.06 

1.98 

1.50  . 

Minor  Production  Advance 

2 

1.02 

1.45 

1.24  ' 

1 

Substantial  Production  Advance 

6 

1.00 

1.98 

1.50 

.711 

Kruskal-Wallis 

Extensive  Production  Advance 

3 

1.06 

1.73 

1.33  - 

1 

DevelopiProd.  Overlap  Subsample  12 
Remainder  of  Sample  7 

1.01 

1.00 

2.45 

2.82 

1.54  1 
1.37  J 

\  .108 

Mann-Whitney 

Since  the  technological  advance  subsample  and  the  development/production 
overlap  subsample  (both  discussed  in  Chapter  m)  are  not  significantly  different 
from  the  remainder  of  the  sample,  as  shown  above  and  in  Figure  IV-6,  they  are 
representative  of  the  entire  sample;  inferences  drawn  about  these  two  subsamples 
should  be  applicable  to  the  entire  sample.  Since  the  resource  requirements  were 
obtained  from  the  same  representative  subsampie  of  munitions  as  were  the 
required  levels  of  technological  advance,  any  inferences  drawn  about  the  resource 
requirements  subsample  should  also  be  applicable  to  the  entire  sample. 


IV- 11.  Spearman  rank  correlations  of  development  schedule  growth  factors  with  the 
percentage  requirements  for  new  test  equipment,  facilities,  and  tooling  are  shown 
below: 


Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Percentage  Requirements  for  New  Resources 
Test  Equipment 

11 

.000 

100.0% 

Facilities 

11 

+.009 

97.7% 

Tooling 

11 

+.064 

84.0% 

IV- 12.  Spearman  rank  correlations  of  development  schedule  growth  factors  with 
development  start  dates  and  with  the  development/production  overlap  ratios 
described  in  Chapter  in  are  shown  below: 


Sample 

Correlation 

Statistical 

Measure 

Size 

Coefficient 

Significance 

Start  of  Full  Scale  Development 

19 

-.116 

62.3% 

Concurrency  Ratio 

12 

+.119 

69.3% 

IV-13.  Spearman  rank  correlations  of  the  number  of  months  in  advanced  development 
with  the  number  of  months  in  full  scale  development  and  the  development 
schedule  growth  factors  are  shown  below.  Statistical  significance  was  interpreted 
using  the  Table  of  Critical  Values  from  the  Chemical  Rubber  Company  Handbook 
of  Tables  for  Probability  and  Statistics  for  small  sample  sizes. 


Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Number  of  Months  in  Full  Scale  Development 
Development  Schedule  Growth  Factor 

9 

9 

+.125 

+.038 

>10.0% 

>10.0% 

The  data  for  Figure  IV-7  are  shown  below: 

Characteristic 

Sample 

Size 

Low 

High 

Mean 

All-Planned 

19 

29 

654 

158 

All-Actual 

19 

29 

472 

148 

Intercept-Planned 

10 

29 

222 

72 

Intercept- Actual 

10 

29 

179 

88 

Surface  Attack-Planned 

9 

35 

654 

254 

Surface  Attack-Actual 

9 

33 

472 

214 

Air  Launched-PIanned 

12 

30 

241 

74 

Air  Launched- Actual 

12 

33 

229 

90 

Surface  Launched-PIanned 

7 

29 

654 

302 

Surface  Launched- Actual 

7 

29 

470 

246 

Army-Planned 

7 

29 

654 

305 

Army-Actual 

7 

29 

470 

259 

Navy-Planned 

10 

30 

218 

67 

Navy-Actual 

10 

37 

141 

85 

Air  Force-Planned 

2 

35 

169 

102 

Air  Force-Actual 

2 

33 

111 

72 

New-Planned 

10 

45 

654 

258 

New-Actual 

10 

37 

472 

211 

Modified-Planned 

9 

29 

113 

48 

Modified- Actual 

9 

29 

134 

78 

IV- 15.  The  data  for  Figure  IV-8  are  shown  below.  The  Mann- Whitney  test  was 
interpreted  using  the  Table  of  Probabilities  in  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal  sample 
sizes. 


Sample 

Statistical 

Characteristic 

Size 

Low 

High 

Mean 

Significance 

Test 

All 

19 

0.65 

4.10 

1.31 

Intercept 

Surface  Attack 

10 

9 

0.66 

0.65 

4.10 

1.01 

1.68  ' 
0.89  . 

\  .147 

Kruskal- Wallis 

Air  Launched 

Surface  Launched 

12 

7 

0.66 

0.65 

4.10 

1.01 

1.57  ' 
0.85  • 

}  .103 

Kruskal- Wallis 

Navy 

10 

0.65 

4.10 

1.70  ) 

•  .890 

Kruskal -Wallis 

Army 

7 

0.72 

1.01 

0.90  J 

}  .334 

Air  Force 

2 

0.66 

0.94 

0.80 

Mann-Whitney 

New 

Modified 

19 

9 

0.65 

0.94 

1.01 

4.10 

0.84  ^ 
1.83  J 

■  .006 

Kruskal- Wallis 

Operational/Technical 
Requirements  Satisfied 
Operational/Technical 

7 

0.65 

1.50 

0.967 

-  .139 

Kruskal-Wallis 

Requirements  not  Satisfied 

10 

0.72 

4.10 

1.65  j 

IV- 16.  Spearman  rank  correlations  of  development  quantity  growth  factors  with 
development  schedule  growth  factors  and  full  scale  development  start  dates  are 
shown  below.  Statistical  significance  for  the  modified  sample  (of  size  9)  was 
interpreted  using  the  Table  of  Critical  Values  from  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small  sample  sizes. 


Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Development  Schedule  Growth- All 

19 

38.4% 

Development  Schedule  Growth-New 

10 

+.073 

82.7% 

Development  Schedule  Growth-Modified 

9 

+.317 

>10.0% 

Full  Scale  Development  Start  Date 

19 

8.8% 

IV- 17.  The  data  for  Figure  IV-9  are  shown  on  the  next  page.  The  Mann- Whitney  tests 
were  interpreted  using  the  Tables  of  Probabilities  and  Critical  Values  in  the 
Chemical  Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for 
small,  unequal  sample  sizes. 


Sample 

Signifi- 

Initiative 

Size 

Low 

High 

Mean  canoe  Test 

With  Advanced  Development  Phase 

12 

0.6S 

1.01 

}  .003  Kruskal-Wallit 

Without  Advanced  Development  Phase 

7 

1.00 

4.10 

With  Advanced  Development  Prototypes 

10 

0.66 

1.01 

°  }.  .042  Kruskal-Wallit 

Without  Advanced  Development  Prototypes 

8 

0.63 

4.10 

With  Competitive  Advanced  Development 

7 

0.66 

1.01 

0  84  }  ^  Marm- Whitney 

With  Non-competitive  Advanced  Development 

5 

0.65 

1.00 

With  FSD  Subsystem  Competition 

4 

0.94 

1.01 

}  >.200  Mann- Whitney 

Without  FSD  Subsystem  Competition 

14 

0.65 

4.10 

With  Independent  Testing 

3 

0.82 

1.00 

j ' }  >.200  Marm- Whitney 

Without  Independent  Testing 

14 

0.65 

4.10 

With  Independent  Cost  Estimate 

6 

0.65 

1.00 

°  }  .008  Kruskal-WaUis 

Without  Independent  Cost  Estimate 

10 

0.72 

4.10 

With  System  Warranties 

6 

0.78 

3.94 

{'”  }  .436  Kmskal-Wallis 

Without  System  Warranties 

10 

0.66 

1.94 

With  Suby  stem  Warranties 

Without  Subsystem  Warranties 

3 

12 

1.00 

0.72 

1.94 

3.94 

JjJ  }  >.200  Marm-Whitney 

The  sample  of  munitions  was  furthei  stratified,  as  shown  below,  in  an 
unsuccessful  attempt  to  separate  out  the  effects  of  advanced  development  (with  or 
without  prototypes)  from  whether  the  munition  was  new  or  modified.  None  of 
the  tests  are  statistically  significant,  and  the  data  are  not  plotted  in  Figure  IV-9. 


Initiative 

Sample 

Size 

iaOW 

High 

Mean 

Statis¬ 

tical 

Signifi¬ 
cance  Test 

With  Advanced  Development  Phase 

New  Munitions 

10 

0.65 

1.00 

0.84  I 

*  >20.0%  Marm-Whitney 

Modiiied  Munitions 

2 

0.94 

1.00 

0.97  J 

With  Advanced  Development  Prototypes 

New  Munitions 

8 

0.66 

1.01 

0.87  •) 

*  71.2%  Mann- Whitney 

Modified  Munitions 

2 

0.94 

1.00 

0.97  J 

Without  Advanced  Development  Prototypes 

New  Munitions 

1 

0.65 

0.65 

0.65  1 

‘  25.0%  Mann- Whitney 

Modified  Munitions 

7 

1.00 

4.10 

2.07  > 

New  Munitions 

With  Advanced  Development  Prototypes 

8 

0.66 

1.01 

0.87  1 

‘  22.2%  Mann- Whitney 

Without  Advanced  Development  Prototypes 

1 

0.65 

0.65 

0.65  J 

Modified  Munitions 

With  Advanced  Development  Phase 

&  Advanced  Development  Prototypes 
Without  Advanced  Development  Phase 

2 

0.94 

1.00 

0.97 

r  16.7%  Marm-Whitney 

or  Advanced  Development  Prototypes 

7 

1.00 

4.10 

2.07  _ 

1 

IV- 18.  The  data  for  Figure  IV- 10  are  shown  below.  The  Mann- Whitney  test  was 
interpreted  using  the  Table  of  Critical  Values  in  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal  sample 
sizes. 


Sample 

Statistical 

Risk  Factor 

Size 

Low 

High 

Mean 

Significance  Test 

Stable  Threat 

Unstable  Threat 

17 

2 

o  o 

4.19 

1.00 

1.35 

0.97 

}  >.200 

Mann- Whitney 

Technology  Advance  Subsample 

11 

0.65 

1.50 

0.92 

}  .026 

Kruskal- Wallis 

Remainder  of  Sample 

8 

0.72 

4.10 

1.84 

DevelopyProcL  Overlap  Subsample  12 

0.66 

4.10 

1.31 

}  .731 

Kruskal-Wallis 

Remainder  of  Sample 

7 

0.65 

3.94 

1.31 

IV- 19.  The  Spearman  rank  correlation  of  development  quantity  growth  factors  with  the 
development/production  overlap  ratios  described  in  Chapter  III  is  shown  below: 


Sample 

Correlation 

Statistical 

Risk  Measure 

Size 

Coefficient 

Significance 

Concurrency  Ratio 

12 

+.070 

81.7% 

IV-20.  The  cost  per  R&D  round  in  millions  of  1989  dollars  is  the  quotient  of  the  actual  or 
currently  planned  development  costs  in  1989  dollars  shown  in  Table  IV-7  divided 
by  the  actual  or  currently  planned  numbers  of  test  articles  shown  in  Table  IV-6. 

IV-21.  The  planned  and  actual  development  costs  in  1989  dollars  shown  in  Table  IV-7 
for  each  munition  are  based  on  the  then-year  costs  shown  in  the  individual 
munition  case  studies  contained  in  Volume  II  of  this  report  Those  costs  were 
divided  by  escalation  factors  to  obtain  the  development  costs  in  1989  dollars.  The 
escalation  factors  are  from  the  Office  of  the  Assistant  Secretary  of  Defense 
(Comptroller)  and  are  shown  on  the  next  page. 


IV-22 


Designator 

Tide 

Escalation 

Factor 

A/RIM-7E 

Sparrow  HI-B  CW 

0.29805 

AIM-7F 

Sparrow  m  Pulse  Doppler 

0.29805 

A/RIM-7M 

Sparrow  in  Monopulse 

0.56708 

AIM-9L 

Sidewinder 

0.33528 

AIM-9M 

Sidewinder 

1.00000 

AIM-54A 

Phoenix 

0.26134 

AIM-54C 

Phoenix 

0.52415 

AIM- 120 A 

AMRAAM 

0.56798 

HM-92A 

Stinger-Basic 

0.35122 

FIM-92A 

Stinger-POST/RMP 

0.35122 

AGM-65D/F/G 

UR  Maverick 

0.45075 

A/R/UGM-84  A/C/D 

Harpoon 

0.32150 

AGM-88A 

HARM 

0.56708 

AGM-114A/B 

Hellfire 

0.45419 

BGM-71A 

TOW  I 

0.27860 

BGM-71D 

Town 

0.84932 

- 

MLRS 

0.57514 

M-712 

Copperhead  CLGP 

0.45419 

5"  Deadeye  SALOP 

0.52415 

The  data  for  Figure  IV- 12  are  shown  below: 

Sample 

Characteristic 

Size 

Low 

High  Mean 

All-Planned 

19 

927.0  280.8 

All-Actual 

19 

62.1 

1,299.9  379.5 

Intercept-Planned 

10 

927.0  208.2 

Intercept-Actual 

10 

62.1 

1,299.9  342.8 

Surface  Attack-Planned 

9 

846.0  361.5 

Surface  Attack-Actual 

9 

179.6 

Air  Launched-Planned 

12 

927.0  309.8 

Air  Launched- Actual 

12 

62.1 

1,299.9  428.1 

Surface  Launched-Planned 

7 

77.7 

454.3  231.1 

Surface  Launched- Actual 

7 

179.6 

466.0  296.2 

Army-Planned 

7 

77.7 

463.0  271.5 

Army-Actual 

7 

179.6 

Navy-Planned 

40.0 

846.0  228.6 

Navy- Actual 

62.1 

Air  Force-Planned 

2 

221.9 

927.0  574.5 

Air  Force-Actual 

2 

236.7 

1,299.9  768.3 

New-Planned 

New-Actual 

10 

209.7 

1,299.9  553.6 

Modified-Planned 

9 

Modified- Actual 

9 

62.1 

356.3  186.2 

IV-23.  The  data  for  Figure  IV- 13  are  shown  below.  The  Mann- Whitney  test  was 
interpreted  using  the  Table  of  Critical  Values  in  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal  sample 
sizes. 


Characteristic 

Sample 

Size 

Low 

High 

Mean 

Statistical 

Significance  Test 

All 

19 

0.84 

4.89 

1.71 

Intercept 

10 

0.84 

4.89 

2.14  \ 

.072 

Kruskal-Wallis 

Surface  Attack 

9 

1.03 

1.79 

1.22  * 

Air  Launched 

12 

0.84 

4.89 

1.86  \ 

.933 

Kruskal- Wallis 

Surface  Launched 

7 

1.03 

2.34 

1.45  J 

Army 

7 

1.03 

2.34 

1.44  } 

.845 

Kruskal-Wallis 

Navy 

10 

0.84 

4.89 

1.99  ■ 

t 

Air  Force 

2 

1.07 

1.40 

1.24  . 

}  >.200 

Mann- Whitney 

New 

10 

1.03 

1.54 

1.26  \ 

.165 

Kruskal-Wallis 

Modified 

9 

0.84 

4.89 

2.20  J 

Operational/Technical 

Requirements  Satisfied 

7 

0.84 

1.70 

1.27  ' 

Operational/Technical 

-  .329 

Kruskal-Wallis 

Requirements  not  Satisfied  10 

1.03 

4.89 

1.98  _ 

IV-24.  Spearman  rank  correlations  of  development  cost  growth  factors  with  development 
quantity  growth  factors  are  shown  below.  Statistical  significance  for  categories 
with  sample  sizes  of  less  than  10  was  interpreted  using  the  Table  of  Critical 
Values  from  the  Chemical  Rubber  Company  Handbook  of  Tables  for  Probability 
and  Statistics  for  small  sample  sizes. 


Sample 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

All 

19 

4.5% 

Intercept 

10 

+.485 

14.6% 

Surface  Attack 

9 

+.367 

>10.0% 

Air  Launched 

12 

+.542 

7.2% 

Surface  Launched 

7 

+.589 

>10.0% 

New 

10 

+.043 

89.9% 

Modified 

9 

+.692 

5.0% 

IV-25.  Spearman  rank  correlations  of  development  cost  growth  factors  with  full  scale 
development  start  dates  and  development  schedule  growth  factors  are  shown  on 
the  next  page.  Statistical  significance  for  the  modified  sample  (of  size  9)  was 
interpreted  using  the  Table  of  Critical  Values  from  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small  sample  sizes. 


Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Development  Schedule  Growth 

All 

19 

+261 

25.8% 

New 

10 

+.315 

34.4% 

Modifications 

9 

+.467 

>10.0% 

Full  Scale  Development  Start  Date 

19 

-.126 

59.4% 

IV-26.  The  data  for  Figure  IV- 14  are  shown  below.  The  Mann-Whitney  tests  were 
interpreted  using  the  Tables  of  Probability  and  Critical  Values  in  the  Chemical 
Rubber  Company  Handbook  Tables  for  Probability  and  Statistics  for  small, 
unequal  sample  sizes. 


Statis- 

deal 

Sample 

Signifi- 

Initiatives 

Size 

Low 

High 

Mean 

cance  Test 

With  Advanced  Development  Phase 

12 

0.98 

1.54 

1.22 

}>  .011  Kniskal-Wallis 

Without  Advanced  Development  Phase 

7 

0.84 

4.89 

2.54 

With  Advanced  Development  Prototypes 

10 

0.98 

1.54 

1.21 

[  .026  Kruskal-Wallis 

Without  Advanced  Development  Prototypes 

8 

0.84 

4.89 

2.36 

With  Competitive  Advanced  Development 

7 

0.98 

1.42 

1.20 

^  .638  Mann-Whitney 

With  Non-competitive  Advanced  Development 

5 

1.06 

1.54 

1.26 

With  FSD  Subsystem  Competition 

4 

1.06 

1.20 

1.11 

}■  .050  Mann-Whitney 

Without  FSD  Subsystem  Competition 

14 

0.84 

4.89 

1.93 

With  Independent  Testing 

Without  Independent  Testing 

3 

14 

1.07 

0.84 

1.54 

4.89 

1.34 

1.76 

J->.200  Mann-Whitney 

With  Independent  Cost  Estimate 

6 

1.06 

1.42 

1.24 

j.  .664  Kniskal-Wallis 

Without  Independent  Cost  Estimate 

10 

0.84 

4.89 

1.97 

With  System  Warranties 

6 

0.98 

4.27 

1.78 

j.  1.000  Kruskal-Wallis 

Without  System  Warranties 

10 

0.84 

2.34 

1.46 

With  Subystem  Warranties 

3 

1.42 

2.34 

1.93 

|>.100  Mann-Whitney 

Without  Subsystem  Warranties 

12 

0.84 

4.27 

1.51 

Note-26 


The  sample  of  munitions  was  further  stratified,  as  shown  below,  in  an 
unsuccessful  attempt  to  separate  out  the  effects  of  advanced  development  (with  or 
without  prototypes)  from  whether  the  munition  was  new  or  modified  None  of 
the  tests  are  statistically  significant,  and  the  data  are  not  plotted  in  Figure  IV- 14. 


Statis- 

tical 

Sample 

Signifi- 

Initiatives 

Size 

Low 

High 

Mean 

cance  Test 

With  Advanced  Development  Phase 

New  Munitions 

Modified  Munitions 

10 

2 

1.03 

0.98 

1.54 

1.07 

;^}>10.0%  Mann-Whitney 

With  Advanced  Development  Prototypes 

New  Munitions 

Modified  Munitions 

8 

2 

1.03 

0.98 

1.54 

1.07 

1.251 

1.03J 

17.8%  Mann-Whitney 

Without  Advanced  Development  Prototypes 

New  Munitions 

Modified  Munitions 

1 

7 

1.16 

0.84 

1.16 

4.89 

1.161 

2.54J 

50.0%  Mann-Whitney 

New  Munitions 

With  Advanced  Development  Prototypes 

8 

1.03 

1.54 

1.251 

100.0%  Mann-Whitney 

Without  Advanced  Development  Prototypes 

1 

1.16 

1.16 

1.16* 

Modified  Munitions 

With  Advanced  Development  Phase 

&  Advanced  Development  Prototypes 
Without  Advanced  Development  Phase 

2 

0.98 

1.07 

1.03 

.  22.2%  Mann-Whitney 

or  Advanced  Development  Prototypes 

7 

0.84 

4.89 

2.54 

IV-27.  The  data  for  Figure  IV- 15  are  shown  below.  The  Mann- Whitney  test  was 
interpreted  using  the  Table  of  Critical  Values  in  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small  unequal  sample  sizes. 


Sample 

Statistical 

Risk  Factor 

Size 

Low 

High 

Mean 

Significance 

Test 

Stable  Threat 

Unstable  Threat 

17 

2 

0.84 

1.07 

4.89 

1.42 

1.76 

1.25 

}  >.200 

Mann- Whitney 

Technology  Advance  Subs  ample 

11 

1.07 

2.34 

1.47  ' 

)■  .457 

Kruskal-Wallis 

Remainder  of  Sample 

8 

0.84 

4.89 

2.04  • 

Minor  Performance  Advance 

1 

1.09 

1.09 

1.091 

Substantial  Performance  Advance 

4 

1.07 

1.70 

1.40 

.  .395 

Kruskal-Wallis 

Extensive  Performance  Advance 

6 

1.28 

2.34 

1.57  _ 

No  Materials  Advance 

1 

1.67 

1.67 

1.67' 

Minor  Materials  Advance 

1 

1.09 

1.09 

1.09 

-  .470 

Kruskal-Wallis 

Substantial  Materials  Advance 

5 

1.16 

1.70 

1.45 

Extensive  Materials  Advance 

4 

1.07 

2.34 

1.53. 

Minor  Production  Advance 

2 

1.67 

1.70 

1.691 

Substantial  Production  Advance 

6 

1.07 

1.54 

1.29 

-  .134 

Marm-Whitney 

Extensive  Production  Advance 

3 

1.28 

2.34 

1.68 

Procurement  Funding 
Micro-Stability  Subsample 
Remainder  of  Sample 

11 

8 

0.98 

0.84 

2.04 

4.89 

1.38  -i 

2.16  J 

•  .620 

Kruskal-Wallis 

DevelopiProd.  Overlap  Subsample  12 

0.98 

4.89 

1.67  ■) 

.800 

Kruskal-Wallis 

Remainder  of  Sample 

7 

0.84 

4.27 

1.77  J 

Since  the  technological  advance  subsample  and  the  development/production 
overlap  subsample  (both  discussed  in  Chapter  IE)  are  not  significantly  different 
from  die  remainder  of  the  sample,  as  shown  above  and  in  Figure  IV-16,  they  are 
representative  of  the  entire  sample;  inferences  drawn  about  these  two  subsamples 
should  be  applicable  to  the  entire  sample.  Since  the  resource  requirements  were 
obtained  from  the  same  representative  subsample  of  munitions  as  were  the 
required  levels  of  technological  advance,  any  inferences  drawn  about  the  resource 
requirements  subsample  should  also  be  applicable  to  the  entire  sample. 

IV-28.  Spearman  rank  correlations  of  development  cost  growth  factors  with  the 
percentage  requirements  for  new  resources  are  shown  below: 


Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Percentage  Requirements  for  New  Resources 
Test  Equipment 

11 

-.509 

10.7% 

Facilities 

11 

-.627 

4.7% 

Tooling 

11 

-.382 

22.7% 

Note-28 


V-29.  The  Spearman  rank  correlation  of  development  cost  growth  factors  with  the 
development/production  overlap  ratios  described  in  Chapter  m  is  shown  below: 


Sample 

Correlation 

Statistical 

Risk  Measure 

Size 

Coefficient 

Significance 

Concurrency  Ratio 

12 

+.070 

81.7% 

IV-30.  The  production  costs  in  1989  dollars  shown  in  Table  IV- 8  for  each  munition  are 
based  on  the  then-year  costs  shown  in  the  individual  munition  case  studies 
contained  in  Volume  II  of  this  report  Those  costs  were  divided  by  escalation 
factors  to  obtain  the  production  costs  in  1989  dollars.  The  escalation  factors  are 
from  the  Office  of  die  Assistant  Secretary  of  Defense  (Comptroller)  and  are 
shown  below: 


Designator 

Title 

Escalation 

Factor 

A/RIM-7E 

Sparrow  ni-B  CW 

0.25339 

AIM-7F 

Sparrow  m  Pulse  Doppler 

0.25339 

A/RIM-7M 

Sparrow  m  Monopulse 

0.52259 

AIM-9L 

Sidewinder 

0.28822 

AIM-9M 

Sidewinder 

1.00000 

AIM-54A 

Phoenix 

0.21589 

AIM-54C 

Phoenix 

0.47191 

AIM- 120 A 

AMRAAM 

0.52698 

FIM-92A 

Stinger-Basic 

0.30756 

FIM-92A 

Stinger-POST/RMP 

0.30756 

AGM-65DAF/G 

HR  Maverick 

0.39947 

A/R/UGM-84  A/C/D 

Harpoon 

0.27497 

AGM-88A 

HARM 

0.52259 

AGM-114A/B 

Hellfire 

0.58405 

BGM-71A 

TOW  I 

0.28613 

BGM-71D 

Town 

0.84734 

- 

MLRS 

0.52118 

M-712 

Copperhead  CLGP 

0.39724 

— 

5"  Deadeye  SALGP 

0.47191 

IV-31.  The  data  for  Figure  IV- 16  are  shown  below: 


Characteristic 

Sample 

Size 

Low 

High 

Mean 

All-Planned 

12 

2,784 

All-Current 

12 

710 

4,151 

2,354 

Intercept-Planned 

6 

629 

2,049 

1,128 

Intercept-Current 

6 

710 

2,768 

1,644 

Surface  Attack-Planned 

6 

510 

2,784 

1,965 

Surface  Attack-Current 

6 

820 

4,151 

3,064 

Air  Launched-Planned 

9 

510 

2,784 

1,451 

Air  Launcbed-Cunent 

9 

710 

3,866 

2,230 

Surface  Launched-Planned 

3 

1,087 

2,553 

1,833 

Surface  Launched-Cunent 

3 

1,575 

4,151 

2,725 

Army-Planned 

4 

510 

2,553 

1,502 

Army-Current 

4 

820 

4,151 

2,249 

Navy-Planned 

7 

655 

2.784 

1,473 

Navy-Current 

7 

710 

3,866 

2,246 

Air  Force-Planned 

1 

2,241 

2,241 

2,241 

Air  Force-Current 

1 

3,532 

3,532 

3,532 

New-Planned 

6 

510 

2,784 

1,689 

New-Current 

6 

820 

4,151 

2,791 

Modified-Planned 

6 

629 

2,553 

1,404 

Modified-C  urren  t 

6 

710 

3,532 

1,917 

IV-32.  The  data  for  Figure  IV-17  are  shown  below.  The  Mann-Whitney  tests  were 
interpreted  using  the  Tables  of  Probability  and  Critical  Values  in  the  Chemical 
Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for  small, 
unequal  sample  sizes. 


Sample 

Statistical 

Characteristic 

Size 

Low 

High 

Mean 

Significance  Test 

All 

12 

0.96 

2.23 

1.58 

Intercept 

Surface  Attack 

6 

6 

1.01 

0.96 

2.12 

2.23 

1.54  i 
1.62  J 

\  .631 

Mann-Whitney 

Air  Launched 

Surface  Launched 

9 

3 

1.01 

0.96 

2.12 

2.23 

1.59  1 
1.55  J 

\  >.200 

Mann-Whitney 

Army 

4 

0.96 

2.23 

1.56  1 

[  .928 

Mann-Whitney 

Navy 

7 

1.01 

2.12 

1.59  J 

Air  Force 

1 

1.58 

1.58 

1.58 

}  1.000 

Mann-Whitney 

New 

6 

1.35 

2.23 

1.66  I 

.423 

Kruskal-Wallis 

Modified 

6 

0.96 

2.12 

1.50  J 

Operational/Technical 
Requirements  Satisfied 
Operational/Technical 

4 

1.45 

2.23 

1.82  1 

L  .154 

Mann-Whitney 

Requirements  Not  Satisfied  8 

0.96 

2.12 

1.46. 

IV-33. 


The  Spearman  rank  correlation  of  production  cost  growth  factors  with 
development  schedule  growth  factors  is  shown  below: 


Sample 

Correlation 

Statistical 

Measure 

Size 

Coefficient 

Significance 

Development  Schedule  Growth 

12 

+.692 

2.2% 

IV-34.  Spearman  rank  correlations  of  production  cost  growth  factors  with  development 
cost  growth  factors,  development  quantity  growth  factors,  and  production  start 
dates  are  shown  below.  Statistical  significance  for  the  new  and  modified  samples 
(of  size  6)  were  interpreted  using  the  Table  of  Critical  Values  from  the  Chemical 
Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for  small 
sample  sizes. 


Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Development  Quantity  Growth 

All 

12 

-.063 

41.8% 

New 

6 

-.200 

>10.0% 

Modified 

6 

+.372 

>10.0% 

Development  Cost  Growth 

12 

-.105 

72.8% 

Production  Start  Date 

12 

-.357 

23.7% 

IV-35. 


The  data  for  Figure  IV- 18  are  shown  on  the  next  page.  The  Mann- Whitney  tests 
were  interpreted  using  the  Tables  of  Probabilities  and  Critical  Values  in  the 
Chemical  Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for 
small,  unequal  sample  sizes. 


Note-31 


Initiative 

Sample 

Size 

Low 

High  Mean 

Statis¬ 

tical 

Signifi¬ 

cance 

Test 

• 

With  Advanced  Development  Phase 

Without  Advanced  Development  Phase 

8 

4 

1.31 

0.96 

2.23 

2.12 

1.61 

1.53 

}  .808 

Maim- Whitney 

With  Advanced  Development  Prototypes 
Without  Advanced  Development  Prototypes 

7 

4 

1.31 

0.96 

2.23 

2.12 

1.63 

1.53 

).  .788 

Marm- Whitney 

With  Competitive  Advanced  Development 

With  Non-Competitive  Advanced  Development 

4 

4 

1.31 

1.35 

2.23 

1.93 

1.64 

1.58 

J.  1.000 

Mann- Whitney 

With  FSD  Subsystem  Competition 

Without  FSD  Subsystem  Competition 

3 

9 

1.58 

0.96 

1.93 

2.23 

1.71 

1.54 

[.>.200 

Mann- Whitney 

With  Independent  Testing 

Without  Independent  Testing 

3 

8 

1.35 

0.96 

1.58 

2.23 

1.44 

1.65 

j.  .630 

Mann-Whitney 

With  Independent  Cost  Estimate 

Without  Independent  Cost  Estimate 

4 

6 

1.39 

0.96 

2.23 

2.12 

1.70 

1.56 

}.  .610 

Maim- Whitney 

With  Low-Rate  Initial  Production  Phase 
Without  Low-Rate  Initial  Production  Phase 

7 

3 

1.35 

0.96 

2.23 

2.12 

1.73 

1.51 

).  .667 

Mann-Whitney 

With  System  Production  Competition 

Without  System  Production  Competition 

5 

7 

1.01 

0.96 

2.12 

2.23 

1.53 

1.62 

)•  .808 

Mann-Whitney 

With  Subsystem  Production  Competition 
Without  Subsystem  Production  Competition 

8 

4 

0.96 

1.35 

2.12 

2.23 

1.50 

1.74 

}.  .460 

Mann-Whitney 

With  Multi-Year  Production  Contracting 
Without  Multi-Year  Production  Contracting 

3 

9 

0.96 

1.01 

2.23 

2.12 

1.59 

1.58 

j.>.200 

Mann-Whitney 

With  System  Warranties 

Without  System  Warranties 

5 

5 

1.31 

0.96 

2.23 

1.93 

1.70 

1.34 

j.  .222 

Mann-Whitney 

With  Subystem  Warranties 

Without  Subsystem  Warranties 

2 

8 

1.01 

0.96 

1.39 

2.23 

1.20 

1.60 

j.  .400 

Mann-Whitney 

• 

With  Foreign  Production 

Without  Foreign  Production 

3 

9 

0.96 

1.01 

2.12 

2.23 

1.51 

1.60 

j.>.200 

Mann-Whitney 

IV-36.  The  data  for  Figure  IV- 19  are  shown  on  the  next  page.  The  Mann- Whitney  tests 
were  interpreted  using  the  Tables  of  Probabilities  and  Critical  Values  in  the 
Chemical  Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for 
small,  unequal  sample  sizes. 


Note-32 


Sample 


Statistical 


Risk  Factor 

Size 

Low 

High 

Mean 

Technology  Advance  Subsample 

8 

0.96 

2.23 

1.57  I 

-  .934 

Remainder  of  Sample 

4 

1.01 

2.12 

1.59  J 

Minor  Performance  Advance 

1 

1.61 

1.61 

1.6ll 

Substantial  Performance  Advance 

3 

0.96 

2.01 

1.52 

-  .806 

Extensive  Performance  Advance 

4 

1.35 

2.23 

1.61 

No  Materials  Advance 

1 

2.01 

2.01 

2.01  ] 

Minor  Materials  Advance 

1 

1.61 

1.61 

1.61 

„  .250 

Substantial  Materials  Advance 

3 

0.96 

1.45 

1.25 

Extensive  Materials  Advance 

3 

1.39 

2.23 

1.73. 

Minor  Production  Advance 

2 

0.96 

2.01 

1.49  *1 

Substantial  Production  Advance 

4 

1.35 

1.61 

1.50 

-  .795 

Extensive  Production  Advance 

2 

1.39 

2.23 

1.81  J 

Production  Funding 

Micro-Stability  Subsample 

10 

0.96 

2.23 

1.55 1 

8 

A 

Remainder  of  Sample 

2 

1.35 

2.12 

1.74  J 

DevelopyProd.  Overlap  Subsample  n 

0.96 

2.23 

1.601 

-  >.200 

Remainder  of  Sample 

1 

1.35 

1.35 

1.35  J 

Test 


Mann- Whitney 


Kruskal-Walis 


.250  Kruskal-Wallis 


Kxuskal-Walis 


Mann-Whitney 


Mann- Whitney 


•  Since  the  technological  advance  subsample,  the  production  funding  micro¬ 

stability  subsample,  and  the  development/production  overlap  subsample  (all 
discussed  in  Chapter  HI)  are  not  significantly  different  from  the  remainder  of  the 
sample,  as  shown  above  and  in  Figure  IV- 19,  they  are  all  representative  of  the 
entire  sample;  inferences  drawn  about  these  three  subsamples  should  be 
applicable  to  the  entire  sample.  Since  the  resource  requirements  were  obtained 
^  from  the  same  representation  subsample  of  munitions  as  were  the  required  levels 

of  technological  advance,  any  inferences  drawn  about  the  resource  requirements 
subsample  should  also  be  applicable  to  the  entire  sample. 


IV-37.  Spearman  rank  correlations  of  production  cost  growth  factors  with  the  percentage 
requirements  for  new  resources,  are  shown  below.  Statistical  significance  was 
interpreted  using  the  Table  of  Critical  Values  from  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small  sample  sizes. 


Risk  Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Percentage  Requirements  for  New  Resources 
Test  Equipment 

8 

-.072 

>10.0% 

Facilities 

8 

+.488 

>10.0% 

Tooling 

8 

+.048 

>10.0% 

Note-33 


IV-38.  Spearman  rank  correlations  of  production  cost  growth  factors  with  the 
procurement  funding  stability  measures  described  in  Chapter  ID  are  shown 
below: 


Risk  Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significant 

Procurement  Funding  Stability 
Macro 

12 

+.546 

7.0% 

Micro 

10 

+.133 

68.9% 

IV-39.  The  Spearman  rank  correlation  of  production  cost  growth  factors  with  the 
development/production  overlap  ratios  described  in  Chapter  HI  is  shown  below: 


Sample 

Correlation 

Statistical 

Risk  Measure 

Size 

Coefficient 

Significant 

Concurrency 

11 

+.600 

5.8% 

IV-40.  The  data  for  Figure  IV-20  are  shown  below: 


Characteristic 

Size 

Sample 

Low 

High 

Mean 

All-Planned 

17 

705 

362,832 

63,843 

All-Actual 

17 

2,285 

452,322 

59,042 

Intercept-Planned 

9 

705 

57,773 

15,968 

Intercept-Actual 

9 

2,285 

50,640 

15,105 

Surface  Attack-Planned 

8 

2,870 

362,832 

117,703 

Surface  Attack-Actual 

8 

3,971 

452,322 

108,471 

Air  Launched-Planned 

11 

705 

57,773 

15,513 

Air  Launched- Actual 

11 

2,285 

60,664 

19,371 

Surface  Launched-Planned 

6 

22,387 

362,832 

152,448 

Surface  Launched- Actual 

6 

8,085 

452,322 

131,773 

Army-Planned 

7 

22,387 

362,832 

134,184 

Army-Actual 

7 

8,085 

452,322 

119,905 

Navy-Planned 

9 

705 

57,773 

12,774 

Navy-Actual 

9 

2,285 

19,961 

11,524 

Air  Force-Planned 

1 

31,078 

31,078 

31,078 

Air  Force-Actual 

1 

60,664 

60,664 

60,664 

New-Planned 

8 

2,870 

362,832 

99,330 

New-Actual 

8 

2,285 

452,322 

86,452 

Modified-Planned 

9 

705 

141,224 

32,299 

Modified- Actual 

9 

3,356 

125,856 

34,678 

IV-41.  The  data  for  Figure  IV-21  are  shown  below.  The  Mann-Whitney  test  was 
interpreted  using  the  Table  of  Critical  Values  in  the  Chemical  Rubber  Company 
•  Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal  sample 

sizes. 


Characteristic 

Sample 

Size 

Low 

High 

Mean 

Statistical 

Significance 

Test 

All 

17 

0.19 

4.76 

1.42 

Intercept 

9 

0.34 

4.76 

LfiST. 

.531 

Kruskal-Wallis 

Surface  Attack 

8 

0.19 

1.98 

1.16J 

Air  Launched 

11 

0.34 

4.76 

1.73\ 

.056 

Kruskal-Wallis 

Surface  Launched 

6 

0.19 

1.90 

0.86J 

Army 

7 

0.19 

1.98 

1.02\ 

.266 

Kruskall- Wallis 

Navy 

9 

0.34 

4.76 

1.67-* 

Air  Force 

1 

1.95 

1.95 

1.95  . 

}•  >.200 

Mann-Whitney 

New 

8 

0.19 

1.98 

0.97  \ 

.112 

Kruskal-Wallis 

Modified 

9 

0.34 

4.76 

1.82J 

Operational/Technical 

Requirements  Satisfied 

6 

0.19 

4.76 

1.41 1 

Operational/Technical 

r 

.233 

Kruskal-Wallis 

Requirements  not  Satisfied  10 

0.59 

2.27 

1.38  J 

IV-42.  The  Spearman  rank  correlation  of  production  quantity  growth  factors  with 
production  cost  growth  factors  is  shown  below: 


Sample  Correlation  Statistical 
Measure  Size  Coefficient  Significance 

Production  Cost  Growth  12  +.014  96.3% 

IV-43.  Spearman  rank  correlations  of  production  quantity  growth  factors  with 
development  cost  growth  factors,  development  schedule  growth  factors,  and 
production  start  dates,  are  shown  below: 


Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Development  Cost  Growth 

17 

+.211 

39.9% 

• 

Development  Schedule  Growth 

17 

+.012 

96.1% 

Production  Start  Date 

17 

+.538 

3.1% 

IV-44.  The  data  for  Figure  IV-22  are  shown  on  the  next  page.  The  Mann-Whitney  test 
was  interpreted  using  the  Table  of  Critical  Values  in  the  Chemical  Rubber 
•  Company  Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal 

size. 


Note-35 


Statis¬ 

tical 

Sample  Signifi-  % 


Initiative 

Size 

Low 

High 

Mean 

canoe 

Test 

With  Low-Rate  Initial  Production  Phase 

8 

0.19 

4.76 

1.69] 

■  .203 

Kruskal- Wallis 

Without  Low-Rate  Initial  Production  Phase 

7 

0.34 

1.90 

0.99  J 

With  System  Production  Competition 

9 

0.59 

2.27 

1.58] 

.048 

Kruskal- Wallis 

Without  System  Production  Competition 

8 

0.19 

4.76 

1.24  J 

With  Subsystem  Production  Competition 

10 

0.89 

4.76 

1.91]. 

.006 

Kruskal- Wallis 

Without  Subsystem  Production  Competition 

6 

0.19 

1.38 

0.64J 

With  Multi-Year  Production  Contracting 

5 

0.19 

1.90 

0.96] 

'  .200 

Mann- Whitney 

Without  Multi-Year  Production  Contracting 

12 

0.34 

4.76 

1.61  J 

With  System  Warranties 

6 

0.19 

4.76 

1.83] 

r  .316 

Kruskal-Wallis 

Without  System  Warranties 

9 

0.34 

2.27 

1.11 J 

With  Subsystem  Warranties 

3 

1.05 

2.27 

L74} 

>.200 

Mann- Whitney 

Without  Subsystem  Warranties 

12 

0.19 

4.76 

1.3 1-* 

With  Foreign  Production 

6 

0.34 

1.25 

0.781 

.021 

Kruskal-Wallis 

Without  Foreign  Production 

11 

0.19 

4.76 

\.lr 

IV-45.  The  data  for  Figure  IV-23  are  shown  below: 


Sample 


Statistical 


Risk  Factor 

Size 

Low 

High 

Mean 

Significance  Test 

Technology  Advance  Subsample 

9 

0.19 

4.76 

1.56] 

1  .995 

Kruskal-Wallis 

Remainder  of  Sample 

8 

0.34 

2.27 

1.26  J 

Minor  Performance  Advance 

1 

1.98 

1.98 

1.98' 

Substantial  Performance  Advance 

3 

0.89 

4.76 

2.53 

■  .200 

Kruskal-Wallis 

Extensive  Performance  Advance 

5 

0.19 

1.90 

0.89 

J 

No  Materials  Advance 

1 

4.76 

4.76 

4.76' 

Minor  Materials  Advance 

1 

1.98 

1.98 

1.98 

,  .J74 

Kruskal-Wallis 

Substantial  Materials  Advance 

3 

0.35 

0.98 

0.74 

Extensive  Materials  Advance 

4 

0.19 

1.95 

1.27. 

Minor  Production  Advance 

2 

0.39 

4.76 

2.83' 

Substantial  Production  Advance 

4 

0.35 

1.98 

1.32 

-  .705 

Kruskal-Wallis 

Extensive  Production  Advance 

3 

0.19 

1.90 

1.05 

Production  Funding 

Micro-Stability  Subsample 

10 

0.19 

4.76 

1.62] 

►  .380 

Kruskal-Wallis 

Remainder  of  Sample 

7 

0.34 

1.66 

1.14  J 

Note- 36 


Since  the  technological  advance  subsample  and  the  production  funding  micro¬ 
stability  subsample  (both  discussed  in  Chapter  m)  are  not  significantly  different 
from  the  remainder  of  the  sample,  as  shown  above  and  in  Figure  IV-23,  they  are 
both  representative  of  the  entire  sample;  inferences  drawn  about  these  two 
subsamples  should  be  applicable  to  the  entire  sample.  Since  the  resource 
requirements  were  obtained  from  the  same  representative  subsample  of  munitions 
as  were  the  required  levels  of  technological  advance,  any  inferences  drawn  about 
the  resource  requirements  subsample  should  also  be  applicable  to  the  entire 
sample. 

IV-46.  Spearman  rank  correlations  of  production  quantity  growth  factors  with  the 
percentage  requirements  for  new  resources  are  shown  below.  Statistical 
significance  was  interpreted  using  the  Table  of  Critical  Values  from  the  Chemical 
Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for  small 
sample  sizes. 


Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Percentage  Requirements  for  New  Resources 
Test  Equipment 

9 

+.108 

>10.0% 

Facilities 

9 

+.108 

>10.0% 

Tooling 

9 

-.183 

>10.0% 

IV-47.  Spearman  rank  correlations  of  production  quantity  growth  factors  with  the 
procurement  funding  stability  measures  described  in  Chapter  in  are  shown 
below: 


Risk  Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Procurement  Funding  Stability 

Macro 

17 

-.106 

67.2% 

Micro 

10 

-.215 

64.5% 

IV-48.  This  is  equivalent  to: 

(Currently  planned  production  span/currently  planned  production  quantity) 
(Originally  planned  production  span/originally  planned  production  quantity) 


and  is  a  measure  of  the  change  in  the  amount  of  time  for  production  of  one  unit. 


IV-49.  The  data  for  Figure  IV-24  are  shown  on  the  next  page.  The  Mann-Whitney  test 
was  interpreted  using  the  Table  of  Critical  Values  in  the  Chemical  Rubber 
Company  Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal 
sample  sizes. 


Sample 

Statistical 

Characteristic 

Size 

Low 

High 

Mean 

Significance 

Test 

All 

17 

0.34 

9.15 

21.9 

Intercept 

9 

0.34 

9.15 

2.16  % 

.149 

Kruskal- Wallis 

Surface  Attack 

8 

1.10 

5.56 

2.22} 

Air  Launched 

Surface  Launched 

11 

6 

0.34 

0.88 

9.25 

5.56 

1.96  •> 
2.61  / 

.209 

Kruskal- Wallis 

Army 

7 

0.88 

5.56 

2.41  , 

.290 

Kruskal- Wallis 

Navy 

9 

0.34 

9.15 

2.14} 

Air  Force 

1 

1.10 

1.10 

1.10 J 

\  >.200 

Marat-Whitney 

New 

8 

1.00 

5.56 

2.40  ■» 

.112 

Kruskal- Wallis 

Modified 

9 

0.34 

9.15 

2.00/ 

Operational/Technical 
Requirements  Satisfied 
Operational/Technical 

6 

0.34 

9.15 

3.391 

.356 

Kruskal-Wallis 

Requirements  not  Satisfied 

10 

0.74 

3.85 

1.60J 

IV-50.  The  Spearman  rank  correlation  of  production  stretchout  factors  with  production 
quantity  growth  factors  is  shown  below: 


Measure 


Sample  Correlation  Statistical 

Size  Coefficient  Significance 


Production  Quantity  Growth  17  -.783  0.2% 


IV-5 1.  The  Spearman  rank  correlation  of  production  stretchout  factors  with  production 

cost  growth  factors  is  shown  below: 

Sample  Correlation  Statistical 

Measure  Size  Coefficient  Significance 

Production  Cost  Growth  12  +.378  21.0%  ^ 


A  linear  regression  equation  was  also  calculated,  and  is  shown  below,  with  the 
values  of  the  t-statistics  shown  in  parentheses  below  the  coefficients. 

Production  Cost  Growth  -  1.29  +  .176  production  stretchout 

(7.67)  (2.16) 

Unfortunately,  the  data  do  not  even  come  close  to  satisfying  the  regression 
analysis  requirements  for  Gaussian  (or  normal,  i.e.,  bell-shaped)  distributions  of 
residuals.  Part  of  the  problem  could  be  the  small  sample  size. 


Note-38 


IV-52.  The  Spearman  rank  correlation  of  production  stretchout  factors  with  production 
start  dates  is  shown  below: 


Measure 


Sample  Correlation  Statistical 

Size  Coefficient  Significance 


Production  Start  Date 


17  -.433  8.4% 


IV-53.  The  data  for  Figure  IV-25  are  shown  below.  The  Mann-Whitney  test  was 
interpreted  using  the  Table  of  Critical  Values  in  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small,  unequal  sample 
sizes. 


Statis- 

deal 

Sample 

Signifi- 

Initiative 

Size 

Low 

High 

Mean 

canoe 

Test 

With  Low-Rate  Initial  Production  Phase 

8 

0.34 

5.56 

1.74-1 

.325 

Kruskal-WaUis 

Without  Low-Rate  Initial  Production  Phase 

7 

0.74 

9.15 

3.01  J 

With  System  Production  Competition 

9 

0.74 

3.85 

1.491 

.211 

Kniskal- Wallis 

Without  System  Production  Competition 

8 

0.34 

9.15 

2.97  J 

With  Subsystem  Production  Competition 

10 

0.34 

2.25 

1.14-1 

.  .017 

Kruskal-Wallis 

Without  Subsystem  Production  Competition 

6 

1.00 

9.15 

4.11  j 

With  Multi-Year  Production  Contracting 
Without  Multi-Year  Production  Contracting 

5 

12 

0.88 

0.34 

5.56 

9.15 

2.561 
2.03  J 

>.200 

Mann-Whitney 

With  System  Warranties 

6 

0.34 

5.56 

1.711 

.316 

Kruskal-Wallis 

Without  System  Warranties 

9 

0.88 

9.15 

2.61  J 

With  Subsystem  Warranties 

3 

0.88 

1.31 

l09) 

>.200 

Mann-Whitney 

Without  Subsystem  Warranties 

12 

0.34 

9.15 

2.55  J 

With  Foreign  Production 

6 

1.19 

9.15 

3.44i 

.  .018 

Kruskal-Wallis 

Without  Foreign  Production 

11 

0.34 

5.56 

1.5 1J 

IV-54.  The  data  for  Figure  IV-26  are  shown  below: 


Sample  Statistical 


Risk  Factor 

Size 

Low 

High 

Mean 

Technical  Advance  Subsample 

9 

0.34 

5.56 

1.731 

\  .500 

Remainder  of  Sample 

8 

0.74 

9.15 

2.70 J 

Minor  Performance  Advance 

1 

1.20 

1.20 

1.20' 

1 

Substantial  Performance  Advance 

3 

0.34 

1.31 

0.92 

[■  .645 

Extensive  Performance  Advance 

3 

0.88 

5.56 

2.32. 

1 

No  Materials  Advance 

1 

0.34 

0.34 

0.34' 

Minor  Materials  Advance 

1 

1.20 

1.20 

1.20 

►  .474 

Substantial  Materials  Advance 

4 

1.00 

2.86 

1.72 

Extensive  Materials  Advance 

3 

0.88 

5.56 

2.21 

* 

Minor  Production  Advance 

2 

0.34 

1.31 

0.831 

Substantial  Production  Advance 

4 

1.00 

2.86 

1.54 

.705 

Extensive  Production  Advance 

3 

0.88 

5.56 

2.58J 

Procurement  Funding 

Micro-Stability  Subsample 

10 

0.34 

5.56 

i.8n 

\  .922 

Remainder  of  Sample 

7 

0.74 

9.15 

2.72  J 

DevelopyProd.  Overlap  Subsample  n 

0.34 

5.56 

1.85' 

f  .615 

Remainder  of  Sample 

6 

0.74 

9.15 

2.80- 

Test 

Kruskal- Wallis 

Kruskal- Wallis  # 

Kruskal- Wallis 

Kruskal- Wallis 

Kruskal- Wallis 

Kruskal-Wallis 


IV-55.  Spearman  rank  correlations  of  production  stretchout  factors  with  percentage 
requirements  for  new  resources  are  shown  below.  Statistical  significance  was 
interpreted  using  the  Table  of  Critical  Values  from  the  Chemical  Rubber  Company 
Handbook  of  Tables  for  Probability  and  Statistics  for  small  sample  sizes. 


Risk  Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

• 

Percentage  Requirements  for  New  Resources 
Test  Equipment 

9 

+.108 

>10.0% 

Facilities 

9 

+.388 

>10.0% 

Tooling 

9 

+.421 

>10.0% 

IV-56.  Spearman  rank  correlations  of  production  stretchout  factors  with  the  procurement 
funding  stability  measures  and  development/production  overlap  ratios  described 
in  Chapter  HI  are  shown  below: 


Risk  Measure 

Sample 

Size 

Correlation 

Coefficient 

Statistical 

Significance 

Procurement  Funding  Stability 

Macro 

17 

+.148 

55.3% 

Micro 

10 

+.094 

77.1% 

Concurrency 

11 

+.177 

57.5% 

Note-40 


IV-57.  The  1989-doIlar  total  program  cost  entries  in  Table  IV- 22  are  based  on  the  then- 
year  development  and  production  costs  shown  in  the  individual  munitions  case 
studies  contained  in  Volume  II  of  this  report.  Those  costs  were  divided  by 
escalation  factors  to  obtain  the  1989-dollar  costs  for  development  and  production. 
The  total  program  cost  is  the  sum  of  the  development  and  production  costs.  The 
escalation  factors  for  development  costs  are  shown  in  Note  IV-21,  and  the 
escalation  factors  for  production  costs  are  shown  in  Note  IV-30. 

IV-58.  The  data  for  Figure  IV-27  are  shown  below: 


Characteristic 

Sample 

Size 

Low 

High 

Mean 

All-Planned 

12 

695 

1,812 

All-Actual 

12 

844 

4,463 

2,705 

Intercept-Planned 

6 

695 

2,409 

1,281 

Intercept-Actual 

6 

844 

3,320 

1,898 

Surface  Attack-Planned 

6 

973 

3,184 

2,343 

Surface  Attack-Actual 

6 

1,326 

4,463 

3,511 

Air  Launched-Planned 

9 

695 

3,184 

1,744 

Air  Launched- Actual 

9 

844 

4,463 

2,610 

Surface  Launched-Planned 

3 

1,304 

2,659 

2,017 

Surface  Launched- Actual 

3 

1,892 

4,447 

2,990 

Army-Planned 

4 

973 

2,659 

1,756 

Army-Actual 

4 

1,326 

4,447 

2,474 

Navy-Planned 

7 

695 

3,184 

1,751 

Navy-Actual 

7 

844 

4,463 

2,628 

Air  Face-Planned 

1 

2,463 

2,463 

2,463 

Air  Force-Actual 

1 

3,768 

3,768 

3,768 

New-Planned 

6 

973 

3,184 

2,108 

New-Actual 

6 

1,326 

4,463 

3,314 

Modified-Planned 

6 

695 

2,659 

1,516 

Modified- Actual 

6 

844 

3,768 

2,096 

The  data  for  Figure  IV-29  are  shown  on  the  next  page.  The  Mann-Whitney  tests 
were  interpreted  using  the  Tables  of  Probabilities  and  Critical  Values  in  the 
Chemical  Rubber  Company  Handbook  of  Tables  for  Probability  and  Statistics  for 
small,  unequal  sample  sizes. 


IV-59. 


Sample 

Statistical 

Characteristic 

Size 

Low 

High 

Mean 

Significance 

Test 

All 

12 

0.99 

2.31 

1.54 

Intercept 

6 

1.10 

2.31 

1.58] 

\  .873 

Kruskal-Waliis 

Surface  Attack 

6 

0.99 

2.12 

1.51 J 

Air  Launched 
Surface  Launched 

9 

3 

1.10 

0.99 

2.31 

2.12 

1.55] 
1.52  J 

’  >.200 

Mann-Whitney 

Army 

4 

0.99 

2.12 

1.49] 

1.000 

Kruskal-Waliis 

Navy 

7 

1.10 

2.31 

1.58  J 

}  1.000 

Kruskal-Waliis 

Air  Force 

1 

1.52 

1.52 

1.52 

New 

6 

1.38 

2.12 

1.56] 

.630 

Kruskal-Waliis 

Modified 

6 

0.99 

2.31 

1.53  J 

IV-60.  The  Spearman  rank  correlation  of  total  program  cost  growth  factors  with 
production  cost  growth  factors  is  shown  below: 


Sample 

Correlation 

Statistical 

Measure 

Size 

Coefficient 

Significance 

Production  Cost  Growth 

12 

+.962 

0.1% 

IV-61.  The  Spearman  rank  correlation  of  total  program  cost  growth  factors  with 
development  cost  growth  factors  is  shown  below: 


Sample 

Correlation 

Statistical 

Measure 

Size 

Coefficient 

Significance 

Development  Cost  Growth 

12 

-.032 

91.7% 

Note-42 
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TECHNOLOGY  AND  RESOURCES  QUESTIONNAIRE 


The  questionnaire  presented  here  was  prepared  in  order  to  obtain  information  from 
contractors  on  how  changes  in  technology  and  production  capacity  requirements  anticipated 
early  in  the  program  compared  with  what  was  later  actually  required.  Because  of  the  many 
major  subsystems  and  contractors  for  the  munitions  in  the  sample,  attention  was  restricted 
to  the  guidance  and  control  subsystems  only.  The  guidance  and  control  subsystem  requires 
the  largest  share,  by  far,  of  both  the  development  and  procurement  funding  for  most  of  the 
munitions  in  the  sample.  The  questionnaire  was  submitted  to  the  following  producers  of 
guidance  and  control  systems  for  the  munitions  in  the  sample: 

General  Dynamics-  Pomona 

Hughes  Aircraft 

Martin-Marietta 

McDonnell  Douglas 

Raytheon 

Rockwell 

Texas  Instruments 

Responses  were  provided  by  four  of  the  contractors.  The  responses  are  proprietary 
and  are  contained  in  Volume  II,  Appendix  A.  Prior  to  being  used  in  the  analyses  in 
Chapter  III  of  this  report,  the  response  data  were  aggregated  so  as  to  make  them 
unidentifiable  as  to  contractor  or  munition. 


Weapon  = 

Subsystem  s 
Contractor  s 

The  purpose  of  this  questionnaire  is  to  obtain  estimates  of  the  extent  to  which  changes  in  tech¬ 
nology  and  production  capacity  were  required  for  the  major  subsystems  of  the  weapon.  Of  interest 
are  both  what  was  anticipated  early  in  the  program  and  what  was  subsequently  actually  re¬ 
quired.  Please  check  off  the  entry  in  each  column  that  you  think  best  describes  the  anticipated 
and  actual  requirements  for  each  of  the  following: 


Anticipated  Actual 


Technology  embodied  in  the  subsystem,  as  compared  to 
technology  embodied  in  similar  subsystems  that  your 
organization  was  responsible  for: 


1 .  Off  the  shelf,  as  is 


2.  Minor  advances  (<10%)  to  existing  technology 


3.  Substantial  advances  (11-50%)  to  existing  technology 


4.  Extensive  advances  (>50%)  to  existing  technology 


5.  All  new  technology 


Material  used  in  the  subsystem,  as  compared  to  materials  used 
in  similar  subsystems  previously  produced  by  your  organization: 


1.  Use  of  existing  materials 


2.  Minor  use  (<10%)  of  new  materials 


3.  Substantial  use  (11-50%)  of  new  materials 


4.  Extensive  use  (>50%)  of  new  materials 
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Anticipated  Actual 


Production  processes  used  in  manufacturing  the  subsystem 
as  compared  to  production  processes  used  in  similar  sub¬ 
systems  previously  manufactured  by  your  organization: 


1 .  Use  of  existing  production  processes  only 


2.  Minor  use  {<10%,  by  value  added)  of  new  production 
processes 


3.  Substantial  use  (1 1  -50%,  by  value  added)  of  new 
production  processes 


4.  Extensive  use  (>50%,  by  value  added)  of  new  produc¬ 
tion  processes 


5.  All  new  production  processes 


Production  facilities  used  in  manufacturing  the  subsystem: 


1  ■  Use  of  existing  facilities  only 


2.  Augmentation  of  existing  facilities  (please  estimate 
the  percentage  increase) 


3.  All  new  production  facililties 


Production  tooling  used  in  manufacturing  the  subsystem: 


1 .  Use  of  existing  tooling  only 


2.  Augmentation  of  existing  tooling  (please  estimate 
the  percentage  of  $  value  increase) 


3.  All  new  production  tooling 


Test  equipment  used  in  development  and  production  of  the 
subystems 


1  •  Use  of  existing  test  equipment  only 


2.  Augmentation  of  existing  test  equipment  (please  estimate 
the  percentage  of  $  value  increase) 


3.  All  new  test  equipment 
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ACQUISITION  CHECKLIST 


The  acquisition  checklist  is  an  extension  of  the  findings  and  recommendations 
contained  in  [B-l],  augmented  by  decision  criteria  identified  during  the  analyses  for  this 
report  and  identified  in  [B-2  through  B-5].  The  checklist  was  prepared  for  use  by  the  staff 
of  the  Under  Secretary  of  Defense  for  Acquisition.  It  is  in  the  form  of  questions  to  be 
answered  at  the  time  a  program  is  initiated  and  prior  to  approvals  at  Milestones  I,  n,  IIIA, 
and  11113.  The  checklist  has  been  formulated  as  questions  to  be  answered,  rather  than  as 
conditions  to  be  acknowledged  with  "yes"  or  "no"  answers.  Many  checklists  are  in  the 
form  of  conditions  to  be  acknowledged  with  "yes"  or  "no"  answers;  the  most  common 
example  is  the  aircraft  pilot's  checklist.  That  type  of  checklist  is  most  applicable  to 
situations  where  conditions  can  be  easily  observed  or  measured  and  a  simple  annotation  can 
show  that  the  condition  has  been  satisfied.  In  a  more  complex  situation  such  as  a  weapon 
acquisition  program,  such  a  checklist  is  too  easily  susceptible  to  superficial  completion. 
Instead,  this  checklist  is  in  a  format  more  similar  to  the  guides  used  in  inspecting  military 
units  with  questions  that  require  substantive  answers. 

A.  OVERALL  PROGRAM  MANAGEMENT 

The  questions  to  be  answered  at  the  time  the  acquisition  program  is  initiated  are 
concerned  with  the  qualifications  and  capabilities  of  the  program  management  staff. 
Although  the  appointment  of  a  Program  Manager  and  selection  of  his  staff  is  a  service 
prerogative,  an  understanding  of  the  talents  needed  for  successful  program  management  is 
necessary,  and  selection  of  those  talents  should  be  checked  in  OSD. 

1 .  What  are  the  demonstrated  leadership  abilities  of  the  program  manager  in: 

a.  Directing  his  subordinates  to  effectively  accomplish  the  assigned  mission? 

b.  Working  effectively  with  appropriate  personnel  from: 

(1)  potential  user  organizations? 

(2)  relevant  intelligence  agencies? 

(3)  superiors  in  the  defense  acquisition  hierarchy? 


(4)  congressional  bodies  and  staffs? 

(5)  contracting  companies? 

2 .  What  levels  of  experience  are  available  on  the  Program  Manager’s  staff  in: 

a.  Military  operations  with  similar  weapons? 

b .  Management  of  other  acquisition  programs  of  similar  difficulty? 

3 .  What  analytic  skills  are  available  on  the  Program  Manager's  staff  for 

a.  Parametric  and  tradeoff  analyses? 

b.  Cost  estimating? 

c.  Project  planning  and  scheduling? 

d.  Test  planning  and  analyses? 

4.  What  technical  engineering  skills  are  available  on  the  Program  Manager's  staff 
for: 

a.  Electronics? 

b.  Software? 

c.  Explosives? 

d.  Propulsion? 

e.  Airframe  and  structural? 

f.  Manufacturing  technology? 

B .  THREAT/REQUIREMENT/TECHNOLOGY 

The  following  questions  should  be  answered  when  the  project  is  initiated,  and 
should  be  reanswered  at  each  of  Milestones  I,  D,  IIIA,  and  IIIB,  (i.e.,  before  approval  to 
commence  concept  demonstration  and  validation,  full  scale  development,  low-rate  initial 
production,  and  full-rate  production). 

1 .  How  do  the  Program  Manager,  users,  and  contractors)  interpret  the  threat  and 
how  the  threat  is  likely  to  change  in  the  future  in  terms  of: 

a.  Target  characteristics? 

b .  Target  environment? 

c.  Numbers  of  targets? 

d.  Possible  countermeasures  against  the  weapon? 

2 .  What  are  the  reactions  of  the  user  community  to  the  program? 


3.  Are  the  system  concept,  design,  cost,  and  schedule  for  development,  testing, 
and  production  consistent  with  historical  data  from  other  similar  systems,  and 
if  not,  why? 

4.  What  are  the  current  initial  operational  capability  (IOC)  estimates  of 
complimentary  systems  that  will  be  used  with  the  weapons  (in  particular, 
platforms,  target  acquisition  systems,  and  target  designation  systems),  and  are 
these  consistent  with  the  current  IOC  estimate  for  the  weapon: 

a.  If  the  weapon  IOC  is  on  the  critical  path,  what  would  be  the  cost  increase 
(expected  and  variance)  to  achieve  the  same  IOC  as  the  next  critical 
complementary  system? 

b.  If  the  weapon  IOC  is  on  a  slack  path,  what  would  be  the  cost  savings 
(expected  and  variance)  from  eliminating  the  slack? 

5.  What  are  the  potential  contractors'  demonstrated  capabilities  to  complete 
projects  requiring  similar  technology  and  advances  in  technology  within 
performance,  schedule,  and  cost  requirements  for  each  subsequent  phase  of  the 
program? 

6.  How  have  the  potential  contractors  performed  in  management  of  high-risk 
programs? 

7.  What  augmentation  of  contractor  and  government-owned  facilities  will  be 
required  for  each  subsequent  phase  of  the  program? 

8.  What  augmentation  of  equipment  and  tooling  will  be  required  for  each 
subsequent  phase  of  the  program;  what  are  the  sources  of  this  equipment  and 
tooling;  will  this  program  significantly  affect  the  availability  and  costs  of  this 
equipment  and  tooling;  and  what  will  be  the  effects  on  other  programs? 

9.  What  augmentation  of  critically  skilled  personnel  will  be  required  for  each 
subsequent  phase  of  the  program;  where  will  they  come  from;  what  special 
training  will  be  required;  and  what  will  be  the  effect  on  other  programs? 

10.  What  cost  inflation  rates  have  been  allowed  for? 

C .  MILESTONE  I 

The  purposes  of  the  next  group  of  questions  are  to  ensure  that  the  technology  to  be 
embodied  in  the  weapon  and  the  technology  to  be  used  in  producing  the  weapon  will  be 
adequately  developed  and  demonstrated  so  that  any  significant  risks  or  uncertainties  of 
design,  production,  and  operational  support  will  be  eliminated  prior  to  engineering 
development 


1 .  What  are  the  concept  alternatives,  and: 

a.  What  alternatives  were  selected  for  demonstration  and  validation,  and 
why? 

b.  What  alternatives  were  rejected  for  demonstration  and  validation,  and 
why? 

c.  What  relevant  work  on  these  alternatives  is  being  done  by  other  military 
services,  government  agencies,  private  industry,  and  research  institutions, 
domestic  and  foreign? 

2 .  For  each  alternative  concept: 

a.  What  are  the  tradeoffs  between  performance  measures? 

b.  How  is  the  expected  ability  to  meet  the  threat  affected  by  varying  the 
performance  measures,  and  what  are  the  levels  of  uncertainly? 

c.  How  would  the  concept  be  used? 

d.  What  are  the  expected  IOC  and  inventory  objective  dates,  what  are  the 
schedule  risks/uncertainties,  and  what  are  the  schedule  drivers? 

e.  What  are  the  expected  development  and  production  costs,  what  are  the 
cost  risks/uncertainties,  and  what  are  the  cost  drivers? 

f.  What  are  the  tradeoffs  between  performance,  schedule,  and  costs? 

g.  What  actions  would  reduce  the  risks/uncertainties,  and  have  contractor(s) 
demonstrated  the  capabilities  to  successfully  accomplish  these  actions? 

h .  What  are  the  alternatives  for  supporting  the  weapon? 

i.  What  are  the  expected  failure  rates,  and  what  are  the  risks/uncertainties? 

j.  What  are  the  expected  support  costs,  and  what  are  the  risks/uncertainties? 

k.  What  tests  and  demonstrations  will  be  required  for  technology 
development,  and  resolution  of  performance  (including  failure  rates), 
production,  schedule,  and  cost  uncertainties? 

l.  At  what  times  and  at  what  throughput  rates  will  testing  be  required? 

m.  Who  will  perform  the  testing,  how  will  their  other  workload  be  affected, 
what  throughput  times  can  they  produce,  and  what  will  be  the  effects  on 
other  acquisition  projects? 

3 .  For  each  alternative,  what  technological  advance  will  be  required  for: 

a.  Each  of  the  subsystems  in  the  weapon? 

b .  The  production  process  for  each  of  the  subsystems  in  the  weapon? 


4 .  When  evaluating  the  alternatives: 

a.  What,  if  any,  prototypes  will  be  required? 

b .  What  experiments  or  tests  will  be  required? 

c.  What  data  will  be  required? 

d.  What  will  the  evaluation  methodology  be? 

5 .  What  are  the  benefits  and  costs  of  sole-source  demonstration  and  validation  vs. 
competitive  demonstration  and  validation? 

D.  MILESTONE  H 

The  purposes  of  the  next  group  of  questions  are  to  ensure  that  prior  to  starting  full 
scale  development  (FSD),  the  technology  to  be  embodied  in  the  weapon  and  the  technology 
to  be  used  in  producing  the  weapon  have  been  adequately  developed  and  demonstrated,  and 
that  technical  and  interface  specifications  have  been  formulated  that  will  enable  a  weapon  to 
be  designed  that  meets  performance,  schedule,  and  cost  requirements. 

1 .  What  are  the  technical  and  system  interface  specifications,  as  determined  from 
the  demonstration  and  validation  phase,  and: 

a.  Are  these  firm  enough  to  allow  full  scale  development  to  proceed? 

b .  Are  these  unnecessarily  rigid  and/or  unproducible? 

2.  What  levels  have  the  risks  or  uncertainties  been  reduced  to  during  the 
demonstration  and  validation  phase,  and  how  was  this  done  for 

a.  Weapon  performance,  schedule,  and  cost? 

b.  Production  technology? 

c .  Support  requirements  ? 

3 .  For  each  category  of  testing  that  was  done  in  the  demonstration  and  validation 
phase: 

a.  How  realistic  was  the  testing,  in  terms  of  the  test  environments  as 
compared  to  potential  operational  environments? 

b.  What  performance  was  demonstrated? 

c.  Was  the  technology  tested  representative  of  the  technology  to  be  used? 


4.  For  each  subsystem  of  the  weapon: 

a.  What  will  the  technology  be? 

b.  What  similar  subsystems  for  other  weapons,  or  what  prototypes  for  this 
subsystem  have  the  contractors)  built? 

c.  What  similar  subsystems  are  already  in  existence,  and  if  they  were 
rejected  for  incorporation  in  the  weapon,  why? 

d.  What  relevant  work  is  being  done  by  other  military  services,  government 
agencies,  private  industry,  and  research  institutions,  domestic  and 
foreign? 

e.  What  materials  will  be  processed  with  what  production  technology  for  the 
critical  areas  of  each  subsystem? 

f .  What  previous  experiences  have  the  contractor(s)  had  in  processing  these 
materials  with  these  production  technologies,  and  what  have  the 
throughput  and  yield  rates  been? 

g .  What  are  the  other  uses  of  these  materials  and  production  processes,  will 
the  program  significantly  affect  their  availabilities  and  costs,  and  what  will 
be  the  effects,  if  any,  on  other  acquisition  programs? 

h.  Are  the  expected  costs  and  productivity  measures  consistent  with  what  is 
being  experienced  with  similar  materials  and  processes  in  other  military 
and  industrial  applications  (both  domestic  and  foreign)? 

i.  How  will  test  articles  be  produced,  in  contrast  to  how  operational 
weapons  will  be  produced? 

5 .  For  each  subsystem  of  the  weapon,  and  for  the  weapon  as  a  whole: 

a.  What  development  testing  will  be  required  during  FSD,  at  what  times,  and 
at  what  rates? 

b.  Who  will  perform  the  testing,  how  will  their  other  workload  be  affected, 
what  throughput  times  can  they  produce,  and  what  will  be  the  effects  on 
other  acquisition  programs? 

c.  How  will  the  operational  suitability  be  demonstrated  to  the  satisfaction  of 
the  potential  users,  OSD,  and  Congress? 

d.  What  other  critical  resources  will  be  required  during  FSD  (e.g.,  super¬ 
computing  for  numerical  analysis  or  finite  element  analysis),  at  what 
times,  and  at  what  rates? 

e.  Who  will  provide  these  other  critical  resources,  how  will  their  other 
workload  be  affected,  what  throughput  times  can  they  produce,  and  what 
will  be  the  effects  on  other  acquisition  programs? 
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6 .  What  design  procedures  have  been  implemented  to  ensure  that  the  weapon  will 
be: 

a.  Testable,  using  the  test  equipment  available  in  the  production,  storage, 
and  operational  environments? 

b.  Producible? 

c .  Reliable  in  both  operational  and  storage  environments? 

d.  Maintainable  in  both  operational  and  storage  environments? 

7 .  What  are  the  benefits  and  costs  of  sole-source  FSD  vs.  competitive  FSD? 

8 .  What  procedures  have  been  implemented  to  obtain  appropriate  documentation 
from  FSD  to  allow  competitive  or  foreign  production  of: 

a.  The  entire  weapon? 

b .  Some  or  all  of  the  subsystems  of  the  weapon? 

E.  MILESTONE  HIA 

The  purposes  of  the  next  group  of  questions  are  to  ensure,  prior  to  starting  limited 
production  and  deployment,  that  a  weapon  has  been  designed  that  meets  operational  and 
performance  requirements,  is  producible  within  quality,  schedule,  and  cost  requirements, 
and  is  supportable  at  reasonable  costs. 

1 .  How  has  it  been  demonstrated  that  the  weapon  is: 

a.  Producible,  within  schedule  and  cost  requirements? 

b .  Supportable,  within  life-cycle-cost  requirements  and  the  planned  logistics 
infrastructure? 

c .  Usable  by  combat  personnel? 

d .  Useful  in  combat  environments  ? 

e.  Storable,  while  awaiting  combat? 

2 .  What  fabrication  methods  will  be  used  in  low-rate  initial  production: 

a.  How  do  these  differ  from  the  fabrication  methods  used  in  development? 

b .  How  have  they  been  demonstrated? 

3 .  What  are  the  lead  times  for,  and  when  will  be  the  availability  of: 

a.  Production  facilities  for  low-rate  initial  production? 

b .  Test  equipment  for  low-rate  initial  production  ? 

c.  Production  tooling  for  low-rate  initial  production? 


d.  Production  materials  for  low-rate  initial  production? 

e .  Production  labor  fra-  low-rate  initial  production? 

f .  Facilities  for  early  operational  support? 

g .  Test  equipment  for  early  operational  support? 

h .  Support  equipments  for  early  operational  support? 

i  Training  of  operations  and  support  personnel  for  early  operational 
support? 

4.  How  much  slack  has  been  provided  in  the  production  schedule  for  unforeseen 
problems? 

5.  What  procedures  have  been  implemented  to  ensure  quality  control  of  the 
production  process  during  low-rate  initial  production? 

6.  What  procedures  have  been  implemented,  and  what  resources  have  been 
obtained  for  early  operational  support? 

7 .  What  procedures  have  been  implemented  to  obtain  data  from  OT&E  and  early 
operational  experience  for  use  in: 

a.  Developing  engineering  fixes? 

b .  Validating  operational  tactics  and  training  requirements? 

c .  Validating  requirements  for  spare  parts  and  maintenance  capacity? 

F.  MILESTONE  IIIB 

The  purposes  of  the  final  group  of  questions  are  to  ensure  that  any  necessary 
engineering  fixes  identified  during  OT&E  and  early  operational  experience  be  incorporated 
into  the  weapon  before  start  of  full-rate  production  (FRP),  and  that  production  be  of  a  high 
quality  and  at  the  lowest  cost  consistent  with  usage  and  inventory  objectives. 

1 .  What  requirements  for  engineering  fixes  have  been  identified  during  OT&E 
and  early  operational  experience,  and  how  are  they  to  be  satisfied;  what  are  the 
operational  and  cost  implications  of  satisfying  them  prior  to  FRP  versus  after 
FRP  has  been  achieved? 

2.  What  fabrication  methods  will  be  used  in  FRP,  and: 

a.  How  do  they  differ  from  the  fabrication  methods  used  in  low-rate  initial 
production  and  development? 

b .  How  have  they  been  demonstrated? 
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What  production  tolerances  have  been  achieved  during  low-rate  initial 
production,  and: 

a.  How  do  actual  tolerances  differ  from  specified  tolerances? 

b .  What  are  the  schedule  and  cost  implications? 

What  yields  and  throughput  times  have  been  achieved  during  low-rate  initial 
production,  and: 

a.  How  do  actual  yields  differ  from  planned  yields? 

b.  How  do  actual  throughput  times  differ  from  planned  throughput  times? 

c.  What  are  the  schedule  and  cost  implications? 

For  each  piece  of  special  tooling  and  test  equipment  used  during  low-rate  initial 
production: 

a.  To  what  extent  have  required  specifications  been  achieved? 

b.  What  is  the  effect  on  yield  and  throughput  rates,  and  on  schedule  and 
costs? 

What  are  the  costs  and  flow  rates  of  alternative  production  processes? 

What  is  the  lowest  cost  production  rate,  and: 

a.  What  degree  of  variability  in  production  rate  is  allowed  at  the  lowest  cost 
rate? 

b.  What  are  the  costs  of  allowing  for  additional  variability  in  production 
rates? 

c.  How  much  slack  is  provided  at  these  rates  for  unforeseen  problems? 

What  are  the  comparative  benefits  and  costs  of: 

a.  Single  source  production? 

b .  Single  source  multiple  year  production? 

c .  Competitive  shared  production  ? 

d.  Winner-take-all  production  competition? 

What  are  the  lead  times  for,  and  when  will  be  the  availability  of: 

a.  Production  facilities  for  FRP? 

b .  Production  equipments  for  FRP? 

c.  Production  material  for  FRP? 

d.  Production  labor  for  FRP? 

e .  Organizational  operating  facilities? 


f .  Organizational  test  equipment? 

g .  Operational  support  equipment  and  tools? 

h.  Organizational  level  spares  inventories? 

i.  Depot  and  intermediate-level  facilities? 

j .  Depot  and  intermediate-level  test  equipment? 

k .  Depot  and  intermediate-level  support  equipment  and  tools? 

l.  Depot  and  intermediate-level  spares  inventories? 

m.  Training  of  operations  and  support  personnel? 

10.  What  procedures  have  been  implemented  by  each  contractor  to  ensure  quality 
control  of  the  production  process  during  FRP? 

11.  What  procedures  have  been  implemented  to  obtain  data  from  operational 
experience  for  use  in: 

a.  Developing  engineering  fixes? 

b .  Validating  operational  tactics  and  training  requirements? 

c .  Validating  requirements  for  spare  parts  and  maintenance  capacity? 
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ABBREVIATIONS 


ABBREVIATIONS 


AD 

advanced  development 

AMRAAM 

Advanced  Medium-Range  Air-to-Air  Missile 

CLGP 

Cannon-Launched  Guided  Projectile 

CW 

continuous  wave 

DAES 

Defense  Acquisition  Executive  Summary 

DCP 

Development  Concept  Paper 

DDR&E 

Director  of  Defense  Research  and  Engineering 

DoD 

Department  of  Defense 

FRP 

full-rate  production 

FSD 

full  scale  development 

HARM 

High-Speed  Anti-Radiation  Missile 

hr 

imaging  infrared 

IOC 

initial  operational  capability 

LCC 

life-cycle  cost 

LRIP 

Low  Rate  Initial  Production 

MLRS 

Multiple-Launch  Rocket  System 

OSD 

Office  of  the  Secretary  of  Defense 

OT&E 

operational  test  and  evaluation 

POST 

Passive  Optical  Seeker  Technique 

RMP 

Reprogrammable  Microprocessor 

SALGP 

Semi-Active  Laser  Guided  Projectile 

SAR 

Selected  Acquisition  Reports 

TOA 

total  obligational  authority 

TOW 

tube-launched,  optically  tracked,  wire-guided 

